Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces PLoP, a cheap, gradient-free Normalized Feature Norm score that picks which module types to LoRA-tune, and shows it beats standard attention-only or MLP-only placement in its experiments.

desk verdict A cheap and plausible method for LoRA placement, but the 'consistently outperforms' claim is carried by test-set-tuned, single-seed comparisons with margins inside seed noise. read the letter →

arxiv 2506.20629 v1 pith:DEEYYNHG submitted 2025-06-25 cs.LG cs.CLstat.ML

Soufiane Hayou , Nikhil Ghosh , Bin Yu This is my paper
classification cs.LGcs.CLstat.ML
keywords LoRAplacementparameter-efficientfinetuningnormalizedfeaturenormsmodule-dataalignmentnormgrowthlargelanguagemodelssupervisedreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that which parts of a pretrained model should receive LoRA adapters can be decided automatically, before training, by measuring the alignment between each module type and the finetuning data. The proposed score, the Normalized Feature Norm (NFN), compares a module's actual output norm on real inputs to its output norm on a random input of the same magnitude; values near one mean little special alignment, and PLoP inserts LoRA into the module types with the lowest scores. The authors ground this in a theoretical analysis showing that feature norms grow during training in modules whose weights align with their inputs, then validate it with supervised finetuning for classification, supervised finetuning for math text generation, and GRPO reinforcement learning for reasoning. In their experiments PLoP beats attention-only, MLP-only, and all-module LoRA in several cases at matched or lower parameter counts, and at worst matches the best baseline. If this is right, practitioners get a nearly free way to replace guesswork about LoRA placement.

What carries the argument

The load-bearing object is the Normalized Feature Norm: for a module with weight $W$ and input feature $z_{\mathrm{in}}(x)$, NFN is $\|W z_{\mathrm{in}}(x)\| / \|W \tilde z_{\mathrm{in}}(x)\|$, where $\tilde z_{\mathrm{in}}(x)$ is an i.i.d. Gaussian vector with the same norm as $z_{\mathrm{in}}(x)$; the denominator strips away the scale of $W$ and of the input, leaving a measure of weight-input alignment. PLoP averages NFN over a batch of task data, aggregates the scores by module type, and places LoRA adapters in the module types with the lowest scores. The theoretical support is the feature-update identity under SignSGD and maximal-update scaling, $W_{t+1} z_{\mathrm{in}} = W_t z_{\mathrm{in}} - \eta n^{-1} \|z_{\mathrm{in}}\|_1 S(dz_{\mathrm{out}})$, which produces norm growth only when the module weight and its input are aligned; Theorem 1 makes this precise for a linear network and predicts quasi-quadratic growth of the normalized feature norm at early training.

What would settle it

Take a fixed model-task pair from the paper, such as Llama-3.2-1B on GSM8K, and exhaustively evaluate every three-module-type LoRA subset with matched parameter counts; the central claim fails if the lowest-NFN combination PLoP recommends does not rank at or near the top in final accuracy.

Watch

Extended reading notes

Core claim

The central discovery is that a module's potential to benefit from LoRA can be read off from its normalized feature norm before finetuning: modules whose real-data features are far less aligned with their weights than a random baseline are the ones where adaptation helps. The authors support this with a theorem in an idealized setting, a wide linear network with one trainable layer, one training point, and SignSGD under maximal-update parameterization, where the normalized feature norm grows quasi-quadratically early in training because alignment between the weight and its input produces a positive update term that a randomized baseline does not receive. This motivates the NFN score and the rule "insert LoRA in the lowest-scoring module types." Empirically, on ANLI classification, MetaMathQA-to-GSM8K text generation, and GRPO for mathematical reasoning, PLoP's chosen module types (for example Value-Out-Down on Qwen3-1.7B) beat attention-only placement by several points, match or beat MLP-only placement, and in several cases outperform all-module LoRA while using fewer trainable parameters.

Load-bearing premise

The method's load-bearing premise is that module types with low Normalized Feature Norm before finetuning are precisely the ones where LoRA adapters will help most, and the paper's theorem does not prove that step, only feature-norm growth in a simplified single-layer, single-example SignSGD setting.

Editorial extensions

If this is right

  • LoRA placement can be chosen at roughly the cost of one batch forward pass, with no full-model gradients, making placement search practical on commodity GPUs.
  • Attention-only LoRA, the default from the original LoRA work, is not the right choice for math reasoning tasks; mixed attention and MLP placements such as Value-Out-Down or Down-Up-Value perform better.
  • PLoP can beat placing adapters in all module types while using roughly 40 to 60 percent of the trainable parameters, for example 43.9 million versus 69.7 million for Qwen3-1.7B.
  • The reverse placement rule, choosing highest-scoring module types, performs worse in the paper's experiments, supporting the interpretation that low alignment rather than high importance marks LoRA potential.
  • NFN rankings are roughly stable across model sizes within the same family, so a ranking computed once on a smaller model may transfer to its larger sibling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested dynamic extension follows from the paper's plateau observation: recompute NFN scores during finetuning and move adapters as alignment saturates, since most feature-norm growth happens in the first few hundred steps.
  • Because specialized models show higher NFN scores than general models on the same task, NFN could serve as a cheap diagnostic of how well a base model is already aligned to a dataset, potentially informing whether to spend budget on SFT or RL.
  • The theoretical bridge from a single-layer, single-datapoint, SignSGD analysis to Adam-trained Transformers with batches is left unproved, so a direct test of whether static low scores are the mechanism rather than a proxy would strengthen or revise the placement rule.
  • Layer-level selection remains an open direction: the paper reports inconsistent layer-level results, suggesting module-type aggregation is the right granularity for now but that better layer-level features could recover finer placement gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PLoP, a method for automatically selecting which module types (e.g., Query, Key, MLP projections) should receive LoRA adapters when finetuning a pretrained model on a given task. The selection is based on a new score, the Normalized Feature Norm (NFN), computed from a small number of forward passes: for each module, the norm of its output feature on task data is divided by a random-baseline norm. The authors motivate the method with a theoretical analysis of feature-norm growth during training (Theorem 1), and they argue that modules with low NFN scores have the most adaptation potential. They present experiments in three settings: supervised finetuning for classification (ANLI), supervised finetuning for text generation (MetaMathQA to GSM8K), and reinforcement learning with GRPO for mathematical reasoning. The central claim is that PLoP consistently outperforms, or at worst matches, common placement strategies such as attention-only or MLP-only placement, often with fewer trainable parameters.

Significance. If validated, PLoP would be a practically useful contribution: it is lightweight (a single forward pass, no full-model gradients), has no fitted parameters, and addresses a real open question in the LoRA literature, where the original LoRA paper and later work give conflicting placement advice. The paper also provides code and reproducible-style experimental details. However, the strength of the empirical evidence does not currently support the headline claim of consistent superiority: the main text-generation comparison selects hyperparameters on the test set with one seed, and all experiments lack uncertainty quantification. The theoretical result (Theorem 1) is for a highly simplified setting and is, as stated, internally inconsistent with its own proof. The core heuristic that low NFN predicts LoRA benefit is asserted rather than derived or directly validated. These issues are fixable, but they are load-bearing for the paper's central claims.

major comments (4)
  1. [Section 4.2, Tables 1–2] The SFT text-generation protocol has a test-set selection flaw. The text states that for each placement strategy the authors sweep the learning rate in {1,2,3,4,5}×10^{-4} and report the result with the best accuracy, where accuracy is GSM8K test accuracy, with no validation split and one seed per run. This means the reported gains are selected on the evaluation metric and could be driven by noise; the margins are small (e.g., Table 1: PLoP r=64 62.0% vs. MLP 63.3%; Table 2: PLoP r=64 75.2% vs. MLP 75.0%). The claim of consistent superiority is therefore not established by these tables. I recommend reporting results with a held-out validation split for hyperparameter selection and multiple seeds with means and variances.
  2. [Section 3, Figure 1, Step 3] The central heuristic that modules with low NFN scores are the best LoRA targets is asserted rather than derived. Theorem 1 shows feature-norm growth during training for a single trainable layer, a single datapoint, and SignSGD; it does not establish that a low static NFN at initialization predicts a larger LoRA benefit after finetuning. The only control for the direction of the heuristic is the PLoP−1 method, but it is evaluated on the same flawed protocol. The paper should either provide a direct derivation linking low NFN to adaptation potential under batch training and Adam, or an explicit validation study (e.g., comparing per-module-type LoRA gains with NFN rankings across models and tasks) that does not rely on the current test-set-selected comparisons.
  3. [Appendix B.2, Theorem 1] The closed-form expression for Γt in Theorem 1 is inconsistent with the recurrence derived in its proof. The proof derives Γ_{t+1} = Γ_t + β²(1+2t), which solves to Γ_t = Γ_0 + β² t², whereas the theorem states Γ_t = Γ_0 + β²(1 + t(t−1)). These differ for t ≥ 2 (e.g., t=2 gives 3β² vs. 4β²). As written, the theorem is false relative to its own proof. This is a correctness issue in the main theoretical result and must be fixed, even if the qualitative "quasi-quadratic growth" conclusion remains.
  4. [Sections 4.1, 4.3, Figure 7, Table 3] All reported experiments are single runs without seeds or error bars. Figure 7 shows single smoothed trajectories, and Appendix C.4 explicitly states that the GRPO runs used a fixed learning rate without a sweep due to resource limits. Table 3 therefore provides no measure of run-to-run variability, so the observed advantages of PLoP over MLP (e.g., 74.52% vs. 73.61% at r=16) could be within noise. The empirical evidence currently supports that PLoP is competitive in several settings, but not that it consistently outperforms, and this should be stated more cautiously.
minor comments (6)
  1. [Abstract] The word "nonconclusive" should be "inconclusive."
  2. [Section 3] The text contains typos: "alignement" should be "alignment," and "These low memory footprint" should be "This low memory footprint."
  3. [Figure 2 caption] The caption contains LaTeX artifacts such as "Feature/uni00A0Norm"; these should be cleaned up.
  4. [Section 2, Notation] The definition of O(d_n) uses "c_n < κ d_n" without absolute values; since sequences are stated to be in R and R_+, this is ambiguous and should be clarified.
  5. [Section 2.1] The notation "n−1∥W2zin2∥2" is confusing because the layer index and the weight matrix name are both "2"; please use distinct subscripts.
  6. [Appendix C.2] The model name "Qwen3.5-0.5B" appears to be inconsistent with the model sizes used in Section 4.1 (Qwen2.5-0.5B or Qwen3-0.6B); please correct.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: NFN-based selection is parameter-free and tested against inverse and fixed baselines; self-citations are peripheral.

full rationale

PLoP's module selection is not circular. NFN(W,D) is computed from the frozen pretrained model and task samples (Definition 1, Section 3), and the selected module types are then evaluated on held-out benchmarks after LoRA finetuning; the scoring rule contains no fitted parameters. The 'low score means more potential for adaptation' direction is an explicit heuristic (Fig. 1, Step 3) and is empirically checked against the inverse rule PLoP-1 and against fixed Attn/MLP baselines, so it is falsifiable rather than true by construction. Theorem 1 is a simplified theoretical support (single trainable layer, single datapoint, SignSGD) and the authors acknowledge the simplification; it neither defines NFN nor entails the placement rule, so the heuristic is not a renamed restatement of the theorem. Self-citations [2], [6], [22], [24] appear in related-work and discussion contexts and are not used to justify the selection rule or to import an unverified uniqueness or ansatz claim. The empirical weakness (learning-rate sweep using GSM8K accuracy with one seed, single GRPO runs) is a correctness and robustness concern, not a reduction of the reported result to its inputs. No circular step meeting the quoted-evidence bar was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The method itself introduces no fitted parameters, but the experimental protocol includes a test-set LR sweep and a hand-chosen selection of three module types. The core heuristic, low NFN equals high adaptation potential, is an assumption, and the toy theory relies on strong simplifications.

free parameters (2)
  • Number of module types selected = 3
    PLoP always selects the three lowest-scoring module types; this number is set by hand to match baseline parameter counts and is not derived from theory.
  • Learning rate in SFT experiments = 1e-4 to 5e-4, chosen per placement by best GSM8K accuracy
    Section 4.2 sweeps learning rate per placement strategy and reports the best test accuracy, which is an optimization against the evaluation set.
assumptions (4)
  • domain assumption Low NFN indicates more potential for adaptation
    Figure 1 and Step 3 of PLoP; not derived from theory, validated only empirically.
  • domain assumption SignSGD captures the relevant behavior of Adam
    Section 2 states 'SignSGD is a nice simplification of Adam'; used to make the analysis tractable.
  • domain assumption Single-layer, single-datapoint, linear network analysis transfers to real Transformers
    Section 2 states 'While this is unrealistic, it provides the right intuition'; the jump to pretrained LLMs with batch training is unverified.
  • domain assumption NFN computed on 100 samples approximates the dataset alignment
    Section 5 example uses --nbsamples 100; no convergence analysis of the estimator is provided.
invented entities (1)
  • Normalized Feature Norm (NFN) independent evidence
    purpose: A forward-pass-only score measuring module-data alignment, used to rank module types for LoRA placement
    NFN makes falsifiable predictions: low-scoring module types should yield better LoRA finetuning, tested in Section 4; the measure itself is newly introduced here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models." pith.science (2026). https://pith.science/paper/DEEYYNHG

@misc{pith2026250620629,
  author       = {Pith},
  title        = {Pith review of: PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEEYYNHG}},
  note         = {Machine review of arXiv:2506.20629}
}
read the original abstract

Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of full finetuning. Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter placement strategy: when using LoRA, practitioners usually pick module types to adapt with LoRA, such as Query and Key modules. Few works have studied the problem of adapter placement, with nonconclusive results: original LoRA paper suggested placing adapters in attention modules, while other works suggested placing them in the MLP modules. Through an intuitive theoretical analysis, we introduce PLoP (Precise LoRA Placement), a lightweight method that allows automatic identification of module types where LoRA adapters should be placed, given a pretrained model and a finetuning task. We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies through comprehensive experiments on supervised finetuning and reinforcement learning for reasoning.

Figures

Figures reproduced from arXiv: 2506.20629 by the authors.

Figure 1
Figure 1. Mechanism of PLoP. We calculate alignment scores called NFN (Normalized Feature Norms), rank them, and pick module types with the lowest alignment scores for LoRA insertion. (see the compute cost paragraph in Section 3 for more details). The mechanism of PLoP is described in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of feature norm growth during training. This shows the fea￾ture norms n −1 ∥W zin∥ 2 for a module W in the model (W ∈ R n×n ). See Section 2.1 for details about the model and training. Mechanisms behind the growth in feature norms. The rea￾son behind this growth in feature norms for certain modules is non-trivial. The naive explanation to this phenomenon is that with training, weight norms grow for some… view at source ↗
Figure 3
Figure 3. Evolution of fea￾ture norms during training for the linear network de￾scribed in Section 2.1. We train the model for 300 steps with Adam. Feature norms for different layers exhibit differ￾ential growth patterns as we train the model. We shifted the curves corresponding to different layers for better vi￾sualization. Consider a three layers linear neural network given by f(x) = W2W1W0x, where x ∈ R d , W0 ∈ R n×d , W1… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: NFN-map for LLama-3.2-1B-Instruct on Math dataset (GSM8K). See Appendix C for NFN-maps of other models. Definition 1 (Normalized Feature Norm (NFN)). Given a pretrained model, a module with weight W in this model, and an input x, we define the Normalized Feature Norm a…
Figure 5
Figure 5. Figure 5: NFN scores aggregated by module type for different models. The scores are for different datasets (math, code, history, and logic). We can also think of the reverse PLoP method where instead of choosing module types with the lowest scores, we choose the ones with the hi…
Figure 6
Figure 6. Figure 6: Module types NFN scores for general and specialized Qwen2.5 models. Specialized models (math, code) are finetuned on task-specific data. Scores are higher with the specialized models. their training data consisted more of sequences similar to general natural language t…
Figure 7
Figure 7. Figure 7: Results of LoRA finetuning on ANLI for different models. We use LoRA rank r = 8 for MLP strategy and adapt r for PLoP and Attn to match number of parameters for fair comparison. All curves are smoothened with EMA(α = 0.8) for better visualization. See Appendix C.3 for …
Figure 8
Figure 8. Figure 8: NFN scores for Qwen3-1.7B 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: NFN scores for Qwen2.5-3B C.7.3 Qwen2.5-1.5B-Instruct 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Layer Index attn.k_proj attn.o_proj attn.q_proj attn.v_proj mlp.down mlp.gate mlp.up Module Type 0.97 1.67 1.43 1.47 1.29 1.38 1.31 1.45 1.36…
Figure 10
Figure 10. Figure 10: NFN scores for Qwen2.5-1.5B 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: NFN scores for Qwen2.5-1.5B-Coder C.7.5 Gemma3-1B-Instruct 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Layer Index attn.k_proj attn.o_proj attn.q_proj attn.v_proj mlp.down mlp.gate mlp.up Module Type 1.17 1.13 1.04 1.18 1.62 1.06 1.41 0.93 1.02…
Figure 12
Figure 12. Figure 12: NFN scores for Gemma3-1B-Instruct 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.

  2. Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards

    cs.LG 2026-07 reject novelty 6.0 of 10

    Non-vacuous PAC-Bayes generalization bounds for billion-parameter RLVR models, obtained by a Gumbel-max reparameterization and aggressive TinyLoRA distillation/quantization, are claimed for four tasks.

  3. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

Reference graph

Works this paper leans on

42 extracted references · 16 canonical work pages · cited by 3 Pith papers

  1. [1]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  2. [2]

    LoRA+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=NEv8YqBROO

    work page 2024

  3. [3]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024. URL https://openreview.net/forum?id=3d5CIRG1n2

    work page 2024

  4. [4]

    RA-LoRA: Rank- adaptive parameter-efficient fine-tuning for accurate 2-bit quantized large language models

    Minsoo Kim, Sihwa Lee, Wonyong Sung, and Jungwook Choi. RA-LoRA: Rank- adaptive parameter-efficient fine-tuning for accurate 2-bit quantized large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024 , pages 15773–15786, Bangkok, Thailand, August 2024. Association f...

    doi:10.18653/v1/ 2024

  5. [5]

    Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision- language models with dynamic rank-selective lora, 2025

    Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, and Dong Gong. Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision- language models with dynamic rank-selective lora, 2025. URL https://arxiv.org/ abs/2412.01004

    arXiv 2025

  6. [6]

    The impact of initialization on lora finetuning dynamics

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 117015–117040. Curran Associates, Inc.,

    work page

  7. [7]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023. URL https://arxiv.org/abs/ 2303.10512

    arXiv 2023

  8. [8]

    Qlora: Effi- cient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Effi- cient finetuning of quantized llms. Advances in neural information processing systems , 36: 10088–10115, 2023

    work page 2023

Show all 42 references
  1. [9]

    Kopiczko, Tijmen Blankevoort, and Yuki M

    Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024. URL https://arxiv.org/abs/2310.11454

    2024 arXiv

  2. [10]

    Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning, 2023

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning, 2023. URL https://arxiv.org/abs/2308.03303

    2023 arXiv

  3. [11]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024. URL https://arxiv.org/ abs/2404.19245

    2024 arXiv

  4. [12]

    Mora: High-rank updating for parameter-efficient fine-tuning, 2024

    Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-efficient fine-tuning, 2024. URL https://arxiv. org/abs/2405.12130. 14

    2024 arXiv

  5. [13]

    A note on lora

    Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora. arXiv preprint arXiv:2404.05086, 2024

    2024 arXiv

  6. [14]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  7. [15]

    The llama 3 herd of models, 2024

    Llama-Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783

    2024

  8. [16]

    Cross-attention is all you need: Adapt- ing pretrained transformers for machine translation

    Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapt- ing pretrained transformers for machine translation. arXiv preprint arXiv:2104.08771, 2021

    2021 arXiv

  9. [17]

    Gradient-based parameter selection for efficient fine-tuning

    Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. Gradient-based parameter selection for efficient fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 28566–28577, 2024

    2024

  10. [18]

    Sensitivity-aware visual parameter-efficient fine-tuning

    Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11825–11835, 2023

    2023

  11. [19]

    Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks,

  12. [20]

    Kingma and Jimmy Ba

    Diederik P . Kingma and Jimmy Ba. Adam: A method for stochastic optimization,

  13. [21]

    signsgd: Compressed optimisation for non-convex problems, 2018

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems, 2018. URL https: //arxiv.org/abs/1802.04434

    2018 arXiv

  14. [22]

    Visualising feature learning in deep neural networks by diagonalizing the forward feature map, 2024

    Yoonsoo Nam, Chris Mingard, Seok Hyeong Lee, Soufiane Hayou, and Ard Louis. Visualising feature learning in deep neural networks by diagonalizing the forward feature map, 2024. URL https://arxiv.org/abs/2410.04264

    2024

  15. [23]

    Implicit regularization via neural feature alignment, 2021

    Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment, 2021. URL https://arxiv.org/abs/2008.00938

    2021 arXiv

  16. [24]

    Feature learning and signal propagation in deep neural networks

    Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machi...

    2022

  17. [25]

    Understanding and minimising outlier features in neural network training, 2024

    Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in neural network training, 2024. URL https://arxiv.org/abs/2405.19279

    2024 arXiv

  18. [26]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  19. [27]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  20. [28]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  21. [29]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: //arxiv.org/abs/2402.03300. 15

    2024 arXiv

  22. [30]

    Qwen3 technical report, April 2025

    Qwen Team. Qwen3 technical report, April 2025. URL https://github.com/QwenLM/ Qwen3/blob/main/Qwen3_Technical_Report.pdf. Released April 29, 2025

    2025

  23. [31]

    Gemma 3 technical report, 2025

    Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503. 19786

    2025

  24. [32]

    Adversarial nli: A new benchmark for natural language understanding

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational ...

    2020

  25. [33]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  26. [34]

    Lillicrap, Kenji Kawaguchi, and Michael Shieh

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P . Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024. URL https://arxiv.org/abs/2405.00451

    2024 arXiv

  27. [35]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  28. [36]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  29. [37]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015. URL https: //arxiv.org/abs/1502.01852

    2015 arXiv

  30. [38]

    On the impact of the activation function on deep neural networks training

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceeding...

    2019

  31. [39]

    G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019. 16 A Additional theoretical details A.1 Infinite-width analysis and µP Scaling re...

    1902 arXiv

  32. [2017]

    URL https://arxiv.org/abs/1412.6980

    arXiv

  33. [2022]

    URL https://arxiv.org/abs/2011.14522

    2011 arXiv

  34. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ d4387c37b3b06e55f86eccdb8cd1f829-Paper-Conference.pdf

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.