Pith. sign in

REVIEW 3 major objections 4 minor 72 references

Advantageous Parameter Expansion Training Makes Better Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read APEX, a stage-wise training method, expands high-impact 'advantageous' heads and channels into low-impact ones; it reports better instruction tuning with 52% of trainable parameters and matched continued pre-training with 33% of the data.

desk verdict Promising instruction-tuning results, but the continued-pretraining headline is undermined by a learning-rate confound and the theory overclaims. read the letter →

arxiv 2505.24241 v1 pith:LMS2ESP7 submitted 2025-05-30 cs.CL

Naibin Gu , Yilong Chen , Zhenyu Zhang , Peng Fu , Zheng Lin , Shuohuan Wang , Yu Sun , Hua Wu
show 2 more authors
Weiping Wang Haifeng Wang
This is my paper
classification cs.CL
keywords advantageousparametersparameterexpansioninstructiontuningcontinuedpre-trainingeffectiverankactivation-basedassessmentlargelanguagemodelstrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

APEX is a training strategy built on the observation that within a pretrained Transformer, a minority of attention heads and feed-forward channels dominate the model's behavior. The paper argues that weaker models concentrate activation in fewer such "advantageous" parameters, while stronger models spread contributions more evenly, and that this share can be deliberately increased. APEX does this in stages: it ranks heads and channels by a Top-K minus Min-K activation score, learns zero-initialized linear operators that map each advantageous slice onto a disadvantageous slice, then fuses the operators back so the architecture is unchanged. If the central claims hold, LLM fine-tuning and continued pre-training can reach or beat full-parameter performance while training substantially fewer parameters, and the paper reports exactly those results: 52% of the parameters outperforming full fine-tuning on instruction tasks, and a 33% data budget matching conventional continued pre-training in perplexity.

What carries the argument

The load-bearing object is the pairwise assessment metric $s^h_{\mathrm{MHA}} = \sum_{\mathcal{D}} \mathbb{I}[A^h_{\mathrm{MHA}} \in \mathrm{Top\text{-}K}] - \sum_{\mathcal{D}} \mathbb{I}[A^h_{\mathrm{MHA}} \in \mathrm{Min\text{-}K}]$, which turns per-sample activation magnitudes into a robust ranking of heads and channels. The expansion operator $\gamma$ is then a zero-initialized linear map $W_{[:,d^N_*]} \mapsto W_{[:,d^P_*]} M + W_{[:,d^N_*]}$, structured as a Monarch product to cut the mapping cost from $O(d^4)$ to $O(d^2)$, and fused into the weight matrix at the end of each stage. The theoretical engine is the effective-rank bound $\mathrm{rank}([W^P, W^N+U]) \ge k + \max(\rho-\delta, s)$, which the paper reads as: improving the underused column space of $W^N$ yields a larger usable rank than merely relying on the already-good $W^P$.

What would settle it

A schedule-matched replication would settle the continued-pre-training claim: run the vanilla baseline with APEX's stage-wise learning-rate decay and no expansion operators, and compare perplexity at 3B and 10B tokens. If the gap to APEX's 5.74 and 5.44 disappears, the expansion is not the cause; if it persists, the confounding is minor.

Watch

Extended reading notes

Core claim

The central claim is that the proportion of advantageous parameters is a controllable training target: a stronger model is one with a larger, more evenly contributing set of high-activation heads and channels, and APEX is a way to grow that set in place. Concretely, APEX partitions each MHA and FFN into advantageous and disadvantageous index sets using relative activation rankings across the training data. It then inserts an expansion operator $\gamma$ that computes $\widetilde{W}_{[:,d^N_*]} = W_{[:,d^P_*]} M + W_{[:,d^N_*]}$ with $M$ zero-initialized, so the starting point matches the original model; the operator is trained over a stage and fused back into $W$. The paper's theoretical analysis bounds the effective rank of the expanded weight matrix as $\mathrm{rank}([W^P, W^N+U]) \ge k + \max(\rho-\delta, s)$, claiming APEX raises the useful rank of underused parameter space. Experiments then show the claimed efficiency: with 52% trainable parameters APEX averages 44.0 across six instruction-tuned benchmarks versus 41.0 for full fine-tuning on a 7B backbone, and in continued pre-training it reaches a lower perplexity than the vanilla baseline after 10B tokens while equalling the baseline's perplexity after only 3B tokens.

Load-bearing premise

APEX's claimed gains stand on the premise that the high-activation heads and channels identified by activation rankings are the right parameters to expand, and that the reported perplexity gap is not an artifact of the unequal learning-rate schedules used in the continued pre-training comparison.

Editorial extensions

If this is right

  • In instruction tuning on a 7B backbone, APEX with 52% trainable parameters averages 44.0 over six benchmarks, beating full fine-tuning (41.0), HFT (42.9), and matching GMT (44.0 at 60% parameters).
  • In continued pre-training, APEX after 3B tokens reaches perplexity 5.74, already below vanilla's 5.83 after 10B tokens; after 10B it reaches 5.44.
  • With only 2.4% trainable parameters, APEX's instruction-tuning average (42.6) surpasses LoRA, DoRA, ReLoRA, and S2FT, and comes close to HFT's 42.9, showing the advantage-expansion mechanism works at PEFT scale too.
  • Ablations attribute 1.1 points on average to the expansion operator and 0.5 points to activation-based assessment over random selection; this is the paper's own decomposition.
  • APEX is plug-and-play: zero-init operators preserve the starting model, fusion restores the original architecture, and the computational overhead in continued pre-training is reported as only 0.7% extra time and 1.01\times FLOPs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference not tested in the paper: if the activation-ranking assessment is the real driver, the same expansion operator could be applied to other structured units, such as layers, mixture-of-experts experts, or KV heads, without changing the algorithm's skeleton.
  • A schedule-controlled replication would separate the expansion benefit from the learning-rate decay; the appendix reports APEX uses stage-wise LR decay from 1e-4 down to 1e-6 while vanilla stays at 1e-4, so the 33%-data claim should be read as conditional on that setup until controlled.
  • The effective-rank story suggests a testable scaling prediction: APEX's benefit should be largest for models whose disadvantageous slices start with low effective rank, and smaller for models already near full rank; the paper does not test this ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces APEX, a staged training strategy that identifies 'advantageous' parameters (MHA heads and FFN channels ranked by activation statistics) and expands them into 'disadvantageous' positions via zero-initialized linear (Monarch) operators that are fused into the weights at the end of each stage. The authors report that APEX outperforms full-parameter tuning on instruction tuning (LLaMA-2-7B and LLaMA-3.1-8B) while training only 52% (or 2.1/2.4%) of parameters, and that in continued pre-training of TinyLLaMA-1.1B APEX reaches a lower perplexity than conventional training at the same budget and matches the 10B baseline at only 3B tokens. A theoretical section argues that APEX improves the effective rank of the weight matrices.

Significance. If the empirical results hold, APEX would be a useful parameter-efficient training technique: it preserves the architecture, adds only about 0.7-1% overhead in continued pre-training, and the ablation and control experiments (Tables 3, 6, 7) give evidence that the benefit comes from the expansion mechanism rather than from simply regularizing activations. The theoretical effective-rank framing is attractive but currently not tight enough to carry the interpretation. The instruction-tuning comparison across two backbones is internally consistent and is the strongest part of the paper; the main weakness is the uncontrolled learning-rate confound in the continued pre-training experiments.

major comments (3)
  1. [§4.2 and Appendix A.3 (Table 2)] The continued pre-training comparison in Table 2 is confounded by unequal learning-rate schedules. Appendix A.3 states that Vanilla CPT trains at a constant 1e-4 learning rate for the full 10B tokens, while APEX starts at 1e-4 in Stage 0 and then decays to 4e-5, 1e-5, 5e-6, and 1e-6 across stages. The PPL gaps in the 3-6B row (5.85 vs 5.55) and the 6-10B row (5.83 vs 5.44) coincide exactly with this aggressive decay, so these gaps cannot be attributed to advantageous-parameter expansion; a lower learning rate alone would typically lower evaluation perplexity in this regime. The 0-3B row uses a matched learning rate and is the only unconfounded comparison, but no error bars or seed-level results are reported, so its statistical significance is unclear. The '33% of data' headline survives this critique only if it relies on that row; the broader convergence-speedup claims from Figure 5 and the later rows of Table 2 do not. Please add a Vanilla run following APEX's LR schedule, or an APEX run with constant LR, and report variance across seeds.
  2. [§3.4, Eq. (15)] The theoretical analysis leading to Eq. (15) does not actually prove that APEX amplifies the effective rank. The key step is the assertion that 'the optimization of M is satisfy to obtain an column index set S such that dim(SP ∩ S(WN+U)[:,S])=0' and that APEX training 'amplifies s through orthogonal subspace learning.' This property is essentially the desired conclusion restated as an assumption; it is not derived from the loss, the update rules, or the expansion operator. In addition, Assumption 1 justifies subspace orthogonality by saying the weight matrix is 'randomly initialized and well pre-trained,' which conflates two different regimes, since pre-trained weights are not random. Please either provide a proof (or a precise condition under which the training objective encourages this orthogonality) or explicitly label this section as a heuristic intuition rather than a theorem.
  3. [§4.1, Figure 4, Table 1] The instruction-tuning evaluation lacks variance estimates and appears to use evaluation-set threshold selection. Figure 4 reports that the threshold K_MHA=K_FFN=18.75% is chosen as 'optimal' on the same evaluation suite used to report Table 1, but no held-out validation for this hyperparameter is described. Many baseline numbers (HFT and GMT, marked with † and ‡) are taken from prior papers rather than re-run under identical conditions, which makes the comparison sensitive to implementation differences. Please report seed-level results with standard deviations for APEX and any re-run baselines, and clarify how the threshold was selected without using the test evaluation results.
minor comments (4)
  1. [Appendix A.3] Appendix A.3 states both 'a 4B tokens training set' and 'a 10B tokens continual pre-training set' in the same paragraph; please clarify which dataset size is used for the results in Table 2 and Figure 5.
  2. [Eq. (3)-(6)] The notation K_MHA and K_FFN is described as a 'proportion threshold' in the text, but the equations treat Top-K and Min-K as counts; please define whether the threshold is a percentage of heads/channels or an absolute number, and keep the usage consistent.
  3. [Figure 4] The figure legend mentions HFT and Full-FT, but the caption only describes the x-axis; please add explicit labels for all curves and include error bars or markers if any seeds were averaged.
  4. [Algorithm 1, line 2] The line 'Initializes MHA ← 0, s FFN ← 0' appears to contain a typo; it should read 's_MHA ← 0, s_FFN ← 0'.

Circularity Check

1 steps flagged · score 4.0 of 10

APEX's benchmark claims are not circular, but the effective-rank theory (Eq. 15) assumes the rank increase it claims to prove; the CPT comparison is also schedule-confounded, though that is a control issue rather than circularity.

  1. self definitional [Section 3.4, Eq. (15)]
    "During the training of APEX, to improve task performance, the optimization of M is satisfy to obtain an column index set S such that dim(SP ∩ S(WN +U)[:,S]) = 0 for the columns |S|=s . As a result, the effective rank of the concatenated matrix, which consists of WP and the columns indexed by S from WN +U , satisfies rank([WP ,(W N +U) [:,S]])≥k+s ."

    The lower bound rank≥k+s is not a derived property of APEX; s is defined as the number of columns of WN+U whose span is disjoint from SP, so k+s follows by the definition of rank. The only way Eq. (15) yields a rank larger than the initial lower bound ρ−δ is the closing assertion that 'APEX training amplifies s through orthogonal subspace learning'. But 'amplifying s' is exactly the claim that APEX increases the effective rank of the expanded matrix. No argument from the APEX update rule, loss, or optimization dynamics is supplied to show s>ρ−δ, so the theoretical 'demonstration' assumes the conclusion it claims to prove. The external benchmark results do not depend on this theoretical step.

full rationale

APEX's headline empirical results are not circular: instruction-tuning gains are measured on MMLU, GSM8K, BBH, TyDiQA, TruthfulQA, and HumanEval against external baselines, and continued pre-training is evaluated by held-out PPL and downstream tasks, so performance is not defined by the method's own activation scores. The self-citations in the paper are not load-bearing: [27] is used alongside an external Monarch-matrix reference [26], and [32] is a published baseline whose results are quoted as reported. The one genuine circular step sits in the theory section: Eq. (15) defines s as the size of a column subset with span disjoint from the advantageous subspace, making rank≥k+s true by construction, and the claim that 'APEX training amplifies s' is exactly the effective-rank increase the theory is supposed to establish. No learning-rate or optimization argument proves s>ρ−δ, so the explanation assumes its own conclusion. This does not infect the benchmark results, which are externally falsifiable. Separately, the continued pre-training comparison in Table 2 is experimentally confounded: Appendix A.3 gives APEX decaying learning rates (4e-5, 1e-5, 5e-6) in later stages while Vanilla stays at 1e-4, so the later PPL gaps cannot be cleanly attributed to APEX; this is an experimental-control limitation, not a circularity. The '33% data' comparison (APEX at 3B with PPL 5.75 vs Vanilla at 10B with PPL 5.83) uses the same Stage-0 learning rate and is not affected by that confound. The score reflects one non-central circular step in the theory, while the central empirical claims remain independent.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The method depends on several domain assumptions about parameter importance and effective rank dynamics, plus hyperparameters selected on the evaluation itself. The continued pre-training result additionally assumes a matched learning rate schedule, which the paper's own appendix contradicts.

free parameters (3)
  • K_MHA and K_FFN selection threshold = 18.75% in instruction tuning, 12.5% in continued pre-training
    Tuned via Fig. 4 in the same evaluation setting; performance changes by 1.1% depending on threshold.
  • Number of stages T = 2 epochs in instruction tuning, 3 stages in continued pre-training
    Chosen based on Fig. 5 and epoch count; T=3 shows better PPL than T=2.
  • Learning rate schedule for APEX stages = 4e-5, 1e-5, 5e-6, 1e-6 after initial 1e-4
    Chosen by hand and confounded with method; no matching schedule for Vanilla.
assumptions (4)
  • domain assumption dim(S_P ∩ S_N) <= delta because weights are randomly initialized and well pre-trained.
    Invoked in Section 3.4 to support the initial subspace orthogonality assumption.
  • domain assumption rank(W_N) = rho << (n-k), i.e., the disadvantageous columns have very low rank.
    Invoked in Section 3.4 to derive the effective rank lower bound.
  • domain assumption Training improves task performance alongside increasing effective rank.
    Imported from reference [28] and used in Section 3.4 to argue that APEX amplifies s.
  • domain assumption Activation magnitude and relative ranking identify parameter importance.
    The whole method relies on this proxy; supported empirically by the mask experiments in Appendix B.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advantageous Parameter Expansion Training Makes Better Large Language Models." pith.science (2026). https://pith.science/paper/LMS2ESP7

@misc{pith2026250524241,
  author       = {Pith},
  title        = {Pith review of: Advantageous Parameter Expansion Training Makes Better Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMS2ESP7}},
  note         = {Machine review of arXiv:2505.24241}
}
read the original abstract

Although scaling up the number of trainable parameters in both pre-training and fine-tuning can effectively improve the performance of large language models, it also leads to increased computational overhead. When delving into the parameter difference, we find that a subset of parameters, termed advantageous parameters, plays a crucial role in determining model performance. Further analysis reveals that stronger models tend to possess more such parameters. In this paper, we propose Advantageous Parameter EXpansion Training (APEX), a method that progressively expands advantageous parameters into the space of disadvantageous ones, thereby increasing their proportion and enhancing training effectiveness. Further theoretical analysis from the perspective of matrix effective rank explains the performance gains of APEX. Extensive experiments on both instruction tuning and continued pre-training demonstrate that, in instruction tuning, APEX outperforms full-parameter tuning while using only 52% of the trainable parameters. In continued pre-training, APEX achieves the same perplexity level as conventional training with just 33% of the training data, and yields significant improvements on downstream tasks.

Figures

Figures reproduced from arXiv: 2505.24241 by the authors.

Figure 1
Figure 1. Comparison of activation distributions between strong and weak models. We use several [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Normalized activation magnitudes of MHA heads and FFN channels. Both are based on LLaMA2-7B and sorted in descending order. Differences in Parameter Advantages. From Figure 2a and 2b, we observe that the magni￾tude of activations exhibits a long-tailed pattern and has significant differences within a mod￾ule. This phenomenon is prevalent in both the MHA and FFN modules across various layers. We further conduct an ex… view at source ↗
Figure 3
Figure 3. An Illustration of APEX. We divide the entire training process into multiple stages. At the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Effect of the number of stages T. 5 10 15 20 25 Layer 3.0 4.0 5.0 6.0 7.0 S t a n d a r d D e via tio n ( ) Full-FT APEX [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: MHA activation distribu￾tions of instruction tuning. 3 6 9 12 15 Layer 1 2 3 4 S t a n d a r d D e via tio n ( ) Vanilla APEX [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 40 canonical work pages

  1. [1]

    Gpt-4 technical report.ArXiv, abs/2303.08774, 2023

    OpenAI. Gpt-4 technical report.ArXiv, abs/2303.08774, 2023

    arXiv 2023

  2. [2]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b.CoRR, abs/2310.06825, 2023

    arXiv

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models.CoRR, abs/2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    arXiv 2023

  4. [4]

    Llama: Open and efficient foundation language models.ArXiv, abs/2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.ArXiv, abs/2302.13971, 2023

    arXiv 2023

  5. [5]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    work page 2025

  6. [6]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    2024

  7. [7]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei...

    2024

  8. [8]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023

    2023

Show all 72 references
  1. [9]

    PiSSA: Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  2. [10]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019

    2019

  3. [11]

    A win-win deal: Towards sparse and robust pre-trained language models

    Yuanxin Liu, Fandong Meng, Zheng Lin, Jiangnan Li, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. A win-win deal: Towards sparse and robust pre-trained language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural I...

    2022

  4. [12]

    The lottery LLM hypothesis, rethinking what abilities should LLM compression preserve? In The Fourth Blogpost Track at ICLR 2025, 2025

    Zhenheng Tang, Xiang Liu, Qian Wang, Peijie Dong, Bingsheng He, Xiaowen Chu, and Bo Li. The lottery LLM hypothesis, rethinking what abilities should LLM compression preserve? In The Fourth Blogpost Track at ICLR 2025, 2025

    2025

  5. [13]

    LLM-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 11

    2023

  6. [14]

    Fluctuation-based adaptive structured pruning for large language models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors,Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirt...

    2024

  7. [15]

    Discovering sparsity allocation for layer-wise pruning of large language models

    Lujun Li, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Syst...

    2024

  8. [16]

    Structured optimal brain pruning for large language models

    Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu. Structured optimal brain pruning for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Langu...

    2024

  9. [17]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  10. [18]

    Dauphin, Angela Fan, Michael Auli, and David Grangier

    Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 20...

    2017

  11. [19]

    Training verifiers to solve math word problems.CoRR, abs/2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.CoRR, abs/2110.14168, 2021

    arXiv

  12. [20]

    Sheared LLaMA: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  13. [21]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024

  14. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  15. [23]

    Massive activations in large language models

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. InFirst Conference on Language Modeling, 2024

    2024

  16. [24]

    Learning to grow pretrained models for efficient transformer training

    Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. InThe Eleventh International Conference on Learning Re...

    2023

  17. [25]

    LEMON: Lossless model expansion

    Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, and Hongxia Yang. LEMON: Lossless model expansion. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  18. [26]

    Compute better spent: Replacing dense layers with structured matrices

    Shikai Qiu, Andres Potapczynski, Marc Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. Compute better spent: Replacing dense layers with structured matrices. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRev...

    2024

  19. [27]

    LEMON: Reviving stronger and smaller LMs from larger LMs with linear parameter fusion

    Yilong Chen, Junyuan Shang, Zhenyu Zhang, Shiyao Cui, Tingwen Liu, Shuohuan Wang, Yu Sun, and Hua Wu. LEMON: Reviving stronger and smaller LMs from larger LMs with linear parameter fusion. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual...

    2024

  20. [28]

    Effective rank and the staircase phenomenon: New insights into neural network training dynamics, 2025

    Jiang Yang, Yuxiang Zhao, and Quanhui Zhu. Effective rank and the staircase phenomenon: New insights into neural network training dynamics, 2025

    2025

  21. [29]

    Goodfellow, and Jonathon Shlens

    Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In Yoshua Bengio and Yann LeCun, editors,4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proc...

    2016

  22. [30]

    bert2BERT: Towards reusable pretrained language models

    Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2BERT: Towards reusable pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual M...

    2022

  23. [31]

    Peters, and Iz Beltagy

    Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy. Staged training for transformer language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors,International Conference on Machine Le...

    2022

  24. [32]

    Hft: Half fine-tuning for large language models, 2024

    Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, and Hua Wu. Hft: Half fine-tuning for large language models, 2024. 14

    2024

  25. [33]

    Enhancing large language model performance with gradient-based parameter selection

    Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, and Peng Cheng. Enhancing large language model performance with gradient-based parameter selection. In Toby Walsh, Julie Shah, and Zico Kolter, editors,AAAI-25, Sponsored by the Association for the Advancement o...

    2025

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  27. [35]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  28. [36]

    Challenging BIG-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoak...

    2023

  29. [37]

    Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

    Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.Transactions of the Association for Computational Lingui...

    2020

  30. [38]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2022

  31. [39]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  32. [40]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th Intern...

    2019

  33. [41]

    DoRA: Weight-decomposed low-rank adaptation

    Shih yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024

    2024

  34. [42]

    ReloRA: High-rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReloRA: High-rank training through low-rank updates. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  35. [43]

    S2ft: Efficient, scalable and generalizable llm fine-tuning by structured sparsity

    Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S2ft: Efficient, scalable and generalizable llm fine-tuning by structured sparsity. InThe 38th Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  36. [44]

    Tinyllama: An open-source small language model, 2024

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024

    2024

  37. [45]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhan...

    2024

  38. [46]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  39. [47]

    The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  40. [48]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124, 2020

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  41. [49]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intellig...

    2020

  42. [50]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors,Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, September 2017. ...

    2017

  43. [51]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. InThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence...

    2020

  44. [52]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors,Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2...

    2019

  45. [53]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and S...

    2019

  46. [54]

    Learning to grow pretrained models for efficient transformer training

    Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogério Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. InThe Eleventh International Conference on Learning Re...

    2023

  47. [55]

    Scaling smart: Accelerating large language model pre-training with small model initialization, 2024

    Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. Scaling smart: Accelerating large language model pre-training with small model initialization, 2024

    2024

  48. [56]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  49. [57]

    Parameter-efficient transfer learning with diff pruning

    Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation...

    2021

  50. [58]

    Training neural networks with fixed sparse masks

    Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    2021

  51. [59]

    LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style...

    1932

  52. [60]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_ llm_leaderboard, 2024

    2024

  53. [61]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2022

  54. [62]

    Openassistant conversations – democratizing large language model alignment, 2023

    Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and A...

    2023

  55. [63]

    Instruction tuning with gpt-4, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023

    2023

  56. [64]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  57. [65]

    Lima: Less is more for alignment, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023

    2023

  58. [66]

    Wizardlm: Empowering large language models to follow complex instructions, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. 17

    2023

  59. [67]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    2023

  60. [68]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.arXiv preprint arXiv:2304.01933, 2023

    Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.arXiv preprint arXiv:2304.01933, 2023

    2023 arXiv

  61. [69]

    composer.https://github.com/mosaicml/composer/, 2021

    The Mosaic ML Team. composer.https://github.com/mosaicml/composer/, 2021

    2021

  62. [70]

    Pythia: A suite for analyzing large language models across training and scaling, 2023

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language mode...

    2023

  63. [71]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  64. [8192]

    [68], we fine-tune them for 3 epochs

    For the parameter-efficient fine-tuning setting, we add the LoRA module to all linear layers for LoRA-based methods and, following Hu et al. [68], we fine-tune them for 3 epochs. All experiments are conducted on 8 Nvidia H100 GPUs (80GB each) Evaluation.We follow the evaluatio...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.