Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Sensitivity-LoRA claims that a Hessian-based sensitivity score computed once before training can allocate LoRA ranks better than uniform or dynamically scheduled ranks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:39 UTC pith:GSMJ5ENP

load-bearing objection A cheap, plausible one-shot LoRA rank allocator whose claimed Hessian-based sensitivity is actually an unvalidated activation-autocorrelation proxy; worth refereeing but needs a major revision. the 4 major comments →

arxiv 2509.09119 v1 pith:GSMJ5ENP submitted 2025-09-11 cs.LG

Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

Hao Zhang , Bo Huang , Zhenjia Li , Xi Xiao , Hui Yi Leong , Zumeng Zhang , Xinwei Long , Tianyang Wang
show 1 more author
Hao Xu
This is my paper
classification cs.LG
keywords Low-Rank AdaptationParameter-Efficient Fine-TuningHessian matrixRank allocationWeight sensitivityFine-tuningGLUE benchmarkLanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning a large language model with LoRA normally gives every weight matrix the same rank, wasting capacity on unimportant matrices and starving important ones. This paper proposes to measure each matrix's importance before training starts using the diagonal of the loss Hessian — approximated cheaply from activation autocorrelations — and to convert those measurements into a one-time rank budget. The method, Sensitivity-LoRA, combines a global trace and two local statistics (top-k average and effective rank) into a single score per matrix, then allocates total rank proportionally. On the GLUE benchmark with RoBERTa-base, it reports an average of 85.94 versus 85.20 for the strongest adaptive baseline, and it reports similar or larger gains on text generation with 7B and 8B models. If the result holds, it shows that a pre-training calibration pass can replace expensive in-training rank scheduling.

Core claim

Sensitivity-LoRA establishes that the sensitivity of a weight matrix to parameter perturbation, measured by the diagonal of the loss function's Hessian at the pretrained initialization, is a sufficient signal for allocating LoRA ranks across the network. The paper defines global sensitivity as the trace of the Hessian and local sensitivity as a blend of the average of the largest Hessian diagonal entries (Top-k) and the minimal number of entries needed to reach a cumulative energy threshold (Effective Rank). These scores are combined with variance-based weights — standard deviation divided by squared mean — so that metrics with more spread dominate the allocation. The resulting per-matrix sc

What carries the argument

The central object is the Hessian-based sensitivity score of a weight matrix. The Hessian H_w is the matrix of second derivatives of the loss with respect to the weights; the method uses its diagonal, under the approximation that the Hessian is diagonally dominant and the network is near a local minimum. Global sensitivity is the trace (sum of diagonal elements); local sensitivity combines the average of the largest k diagonal entries (Top-k) and the minimum number of entries whose cumulative sum reaches a fraction α (Effective Rank). The two metrics are merged with σ/μ² weights, and the final per-matrix score θ_w is normalized across matrices to set each rank r_w = θ_w / Σ θ_w · r_total. Th

Load-bearing premise

The load-bearing premise is that the diagonal of the Hessian of the loss with respect to the weights — computed at the pretrained initialization, under the assumptions that the gradient is zero and that off-diagonal terms are negligible, and approximated by the autocorrelation of activations — faithfully identifies which LoRA matrices will benefit most from a larger rank during training.

What would settle it

Compare Sensitivity-LoRA's allocation to a control that receives the same set of rank values (the same multiset) but assigns them to matrices at random, keeping the total budget fixed. If the random-assignment model matches Sensitivity-LoRA's scores on GLUE and the NLG benchmarks, then the sensitivity signal is not the cause of the gains. A second check: compute sensitivity scores from the autocorrelation proxy and from an exact or block-diagonal Hessian on a small model; if the two produce different rank orderings and the better allocation comes from the exact Hessian, the proxy is the active

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • With the same total rank budget, sensitivity-based allocation raises GLUE average from 85.20 (AdaLoRA) to 85.94 on RoBERTa-base, with the largest single-task gain on RTE (81.81 vs. 79.56).
  • On 7B and 8B language models, the method improves BLEU-4, ROUGE-1, and ROUGE-L on both Magpie-Pro and OpenPlatypus, with the best average score on both backbones.
  • The one-time calibration pass costs about 26 seconds on a 8B model with a full calibration set (and under 10 seconds with 10% of it), so the training-time overhead is negligible compared with AdaLoRA's per-step scheduling.
  • Rank orderings are stable: Kendall's Tau exceeds 0.9 across calibration domains, 0.98 with as little as 10% of calibration data, and 0.98 when the ordering is re-measured after each of 5 epochs, meaning the pre-training allocation does not go stale.
  • The method is not restricted to text: an appendix reports CIDEr/ROUGE-L gains on COCO2017 captioning with a 7B multimodal model, suggesting the sensitivity signal transfers to vision-language adaptation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the rank ordering stays stable across tasks and epochs, the sensitivity signal is likely a property of the pretrained weights and the calibration distribution rather than of the final task. If so, one calibration pass per base model could serve many downstream fine-tunings, amortizing the 25-second cost across all tasks for that model.
  • The variance-weighted combination of metrics (σ/μ²) treats each sensitivity measure as a noisy estimator and upweights the one whose spread across matrices is largest. An equivalent coefficient-of-variation weighting could be tested against a learned or equal-weight blend to see whether the choice of weights, rather than the Hessian signal itself, drives the gains.
  • The Hessian diagonal is approximated by the autocorrelation matrix of activations, a surrogate also used in post-training quantization. If the approximation is the reason the method works, then a rank allocation derived from any low-cost activation-statistic proxy should give similar gains; if not, the second-order loss curvature is doing work the proxy cannot replace.
  • The method allocates ranks before training and never revisits them; the epoch-wise Kendall's Tau suggests the ranking barely changes, but the paper does not test the reverse direction: whether periodically re-running calibration and reallocating ranks would improve results further. That is a natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Sensitivity-LoRA, a parameter-efficient fine-tuning method that assigns LoRA ranks to weight matrices based on sensitivity metrics derived from second-order derivatives (Hessian). The method computes global sensitivity via the Hessian trace and local sensitivity via Top-k and Effective Rank metrics, fuses them using weights of the form sigma/mu^2, and allocates ranks proportionally to the fused scores (Eqs. 8-15). Experiments on GLUE with RoBERTa-base and on NLG tasks with Qwen2.5-7B and LLaMA3.1-8B report consistent improvements over LoRA, AdaLoRA, DyLoRA, and adapter baselines, with low computational overhead. The paper also presents robustness checks using Kendall's tau across domains, calibration set sizes, and training epochs.

Significance. If the empirical results hold, the method would be a useful practical contribution: it provides a static, low-overhead rank-allocation rule that outperforms uniform LoRA and several adaptive baselines across NLU and NLG benchmarks. The stability analysis across domains and calibration sizes is a genuine strength, and the latency/memory comparisons are informative. However, the central theoretical justification is not established: the sensitivity metrics are derived from a Hessian proxy whose connection to the fine-tuning loss curvature is not validated. The paper's contribution therefore currently rests on an unsubstantiated mechanism, although the empirical comparison is potentially reproducible.

major comments (4)
  1. [Section 3.1, Eq. (5)] The derivation drops the first-order term g^T δw by assuming the parameters are at a local minimum (g=0). This is not true at the start of fine-tuning, where the pretrained weights are not at a minimum of the task loss. Consequently, the claimed reduction to Eq. (7) does not justify that diagonal Hessian elements measure sensitivity for the actual fine-tuning setting. The paper should either provide an argument that the first-order term is negligible for the small LoRA perturbations considered, or validate empirically that the proposed sensitivity ordering correlates with actual loss changes under perturbation.
  2. [Appendix E] The Hessian is approximated by the autocorrelation matrix of activations, stated as a surrogate borrowed from GPTQ-style quantization. No derivation shows that this proxy reflects the curvature of the task loss E in Eq. (2). All subsequent metrics (trace, Top-k, Effective Rank), the weights θ_w in Eq. (14), and the ranks in Eq. (15) are computed from this proxy. The robustness checks in Section 4.7 (Kendall's tau) only show that the proxy is stable across domains, epochs, and calibration sizes; they do not show that it is correct. Without validation against exact Hessian diagonals or against perturbation-based sensitivity, the central mechanism is unsupported.
  3. [Abstract and Section 4.6] The paper repeatedly describes the method as 'dynamically allocates ranks,' but the allocation is computed once from a calibration set before training, as stated in Section 4.6 ('Our method allocates the rank before training'). Table 6 shows that the rank ordering remains stable across epochs, which actually confirms that the allocation is static rather than adaptive. This is a conceptual overclaim that affects the framing of the contribution relative to methods such as AdaLoRA, which do reallocate during training. The terminology should be corrected or the method should be extended to genuinely update ranks during training.
  4. [Tables 1-3] All experimental results are reported without error bars, confidence intervals, or multiple seeds. Many differences are small (e.g., average GLUE 85.94 vs. 85.20 for AdaLoRA; NLG average 37.98 vs. 37.39 for Qwen2.5-7B). Without a measure of run-to-run variance, the claim of 'robust effectiveness and stability' is not statistically supported. At minimum, the authors should provide multiple seeds (3-5) and report means and standard deviations for the main results.
minor comments (5)
  1. [Section 3.2.2, Eq. (9)-(10)] The notation lambda^w_i is used for sorted diagonal Hessian elements, but lambda conventionally denotes eigenvalues. Clarify that these are diagonal entries, not eigenvalues, to avoid confusion.
  2. [Section 3.2.2, Eq. (11)] The rationale for the beta/gamma weighting as sigma/mu^2 is stated informally ('larger standard deviation implies more information'). A more precise interpretation (e.g., as a coefficient of variation normalization) and a citation or derivation would help.
  3. [Figure 1] Figure 1 contains multiple duplicated equation blocks and is visually cluttered. Redraw with a cleaner pipeline diagram and place the equations in the main text rather than in the figure.
  4. [Section 2 and References] Several references are incomplete or placeholders, e.g., 'G. Ding et al.' in the references and '?' in Section 2.2 and 2.3. The reference list needs thorough cleanup.
  5. [Appendix B.1] The comparison of the proposed sigma/mu^2 weighting against 0.5/mu is useful, but it is performed on the same GLUE tasks used to select the design. Report whether these parameters were chosen prior to seeing the test results, or validate on a held-out task.

Circularity Check

0 steps flagged

No significant circularity: sensitivity metrics are data-derived, not fitted to target performance.

full rationale

The rank allocation in Eq. (15) is computed from sensitivity metrics S_global (trace of Hessian, Eq. 8), S_Topk (Eq. 9), and S_EffectiveRank (Eq. 10), combined through weights beta and gamma that are defined as sigma/mu^2 of the metric distributions (Eqs. 11, 13). No constant in the method is fitted to maximize the reported GLUE or NLG scores; the hyperparameters (k, alpha) are standard and robustness is checked in Appendix B.2. The Taylor-expansion derivation in Section 3.1 (Eqs. 2-7) is a standard sensitivity argument, and the assumption of a zero gradient at a local minimum is an approximation rather than a circular reduction. Appendix E states that the Hessian is approximated by the autocorrelation matrix of activations; this is an unvalidated surrogate, which is a correctness/evidence concern, but it is not equivalent to the target claim or to the benchmark numbers. The paper's self-citations are background references and are not load-bearing for the rank-allocation equations. The method is evaluated on standard external benchmarks (GLUE, Magpie-Pro, OpenPlatypus), so the reported results are not forced by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim depends on multiple unproven approximations: diagonal dominance, zero-gradient assumption, activation-autocorrelation as a Hessian proxy, and the transferability of static sensitivity. These are domain assumptions, not derived facts.

free parameters (5)
  • alpha (effective rank threshold) = 0.85 (tested also 0.80)
    Hand-chosen threshold in Eq. 10; controls EffectiveRank metric.
  • k (Topk count) = N/2 (half of diagonal elements)
    Hand-chosen count in Eq. 9; controls Topk metric. Tested N/3 as well.
  • beta1, beta2 (local metric weights) = computed as sigma_ST/(mu_ST)^2 and sigma_SE/(mu_SE)^2
    Weighting scheme chosen by hand and validated on the same tasks; not fitted per task.
  • gamma1, gamma2 (global/local fusion weights) = computed as sigma_Sg/(mu_Sg)^2 and sigma_Sl/(mu_Sl)^2
    Chosen by hand; ablation shows it beats equal weights by 0.29 avg GLUE.
  • r_total (average rank per matrix) = 4 for NLU, 8 for NLG
    Standard LoRA rank budget used to match baseline parameter counts.
axioms (4)
  • domain assumption Hessian matrix is diagonally dominant
    Invoked in Eq. 6 (Section 3.1) to ignore off-diagonal terms; cited from pruning literature, but not verified for fine-tuning losses.
  • domain assumption Gradient of the task loss is zero at the point of sensitivity measurement
    Eq. 4 to Eq. 5 (Section 3.1) drops the first-order term by assuming a local minimum; a pre-trained model is not at a minimum of the fine-tuning task loss.
  • ad hoc to paper Activation autocorrelation approximates the loss Hessian
    Appendix E states the Hessian is replaced by the autocorrelation matrix of activations, with no derivation; this is an efficiency surrogate.
  • domain assumption Static pre-training sensitivity predicts optimal rank allocation during fine-tuning
    Section 4.6 and Tables 4-6 show rank-order stability, but stability does not establish that the static allocation maximizes fine-tuning performance.

pith-pipeline@v1.3.0-alltime-deepseek · 18268 in / 13232 out tokens · 132459 ms · 2026-08-04T19:39:39.172652+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models." pith.science (2026). https://pith.science/paper/GSMJ5ENP

@misc{pith2026250909119,
  author       = {Pith},
  title        = {Pith review of: Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSMJ5ENP}},
  note         = {Machine review of arXiv:2509.09119}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.

Figures

Figures reproduced from arXiv: 2509.09119 by Bo Huang, Hao Xu, Hao Zhang, Hui Yi Leong, Tianyang Wang, Xinwei Long, Xi Xiao, Zhenjia Li, Zumeng Zhang.

Figure 1
Figure 1. Figure 1: Pipeline of the Sensitivity-LoRA Method: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of the evaluation results for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The rank allocation for each layer of GPT-2 Large and RoBERTa-base under different rank allocation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of per-step fine-tuning latency [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The Case Study of the GPT-2 Large and RoBERTa-base models. The blue boxes represent the input test data, the green boxes indicate the reference text or ground truth output, and the red boxes highlight the model’s actual output. the dynamic rank allocation method (Sensitivity￾LoRA) on the E2E and SST-2 datasets. For the E2E dataset, the GPT-2 Large model generates fluent and grammatically correct natural la… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

    cs.AI 2026-06 unverdicted novelty 6.0

    MedEvoEval is an executable longitudinal evaluation framework that converts medical cases into action-gated simulated episodes to track how doctor agents evolve decision-making, resource use, and experience across mul...

  2. A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions

    cs.CL 2026-06 unverdicted novelty 5.0

    A3M integrates adaptive DRL, adversarial opponent modeling, and multi-objective rewards to cut regret 30-40% versus baselines while remaining robust to strategy shifts in repeated auctions.

  3. Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models

    cs.CV 2026-06 unverdicted novelty 5.0

    Applies differentiable search over prompt fusion schemes (concatenation, addition, affine, cross-attention) per ViT layer to improve visual prompt tuning, reporting gains across 34 datasets.

  4. EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Control

    cs.CL 2026-06 unverdicted novelty 4.0

    EVLA combines a Unified Co-State Encoder and Electro-aware Structured Reasoning Chain with physics-guided training to produce energy-optimal driving decisions, reporting +5.6% accuracy gains over fine-tuned VLM baseli...

  5. FinInvest-GTCN: Explainable Graph-Temporal-Causal Modeling for Risk-Aware Investment Decision Optimization

    cs.CL 2026-06 unverdicted novelty 4.0

    FinInvest-GTCN combines graph, temporal, and causal networks with meta-causal adaptation to improve risk-adjusted predictions for VC investments, achieving RA-MSE of 2.51 and 18.7% higher simulated returns on propriet...

Reference graph

Works this paper leans on

30 extracted references · 23 linked inside Pith · cited by 5 Pith papers

  1. [4]

    Positive

    We set the average rank of each matrix to 4 for NLU and 8 for NLG. The comparison methods are required to use a similar number of finetuning parameters. The training is performed using the Adam optimizer with a learn- ing rate of 5×10 −4, a batch size of 32 for 10 epochs. B Parameter Analysis B.1 Effectiveness of Allocation Parameter We conduct validation...

    arXiv 2093

  2. [8]

    Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda

    Prefix-tuning: Optimizing continuous prompts for generation tasks.Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021:4582–4597. Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda

    2021

  3. [11]

    arXiv preprint arXiv:2403.16187

    Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403.16187. Yang Luo, Shiru Wang, Jun Liu, Jiaxuan Xiao, Run- dong Xue, Zeyu Zhang, Hao Zhang, Yu Lu, Yang Zhao, and Yutong Xie

    Pith/arXiv arXiv

  4. [12]

    Chenrui Ma, Rongchang Zhao, Xi Xiao, Hongyang Xie, Tianyang Wang, Xiao Wang, Hao Zhang, and Yan- ning Shen

    Pathohr: Breast cancer survival prediction on high-resolution pathological images.arXiv preprint arXiv:2503.17970. Chenrui Ma, Rongchang Zhao, Xi Xiao, Hongyang Xie, Tianyang Wang, Xiao Wang, Hao Zhang, and Yan- ning Shen

    Pith/arXiv arXiv

  5. [13]

    arXiv preprint arXiv:2503.07938

    Cad-vae: Leveraging correlation- aware latents for comprehensive fair disentanglement. arXiv preprint arXiv:2503.07938. Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu

    arXiv

  6. [14]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    Dora: En- hancing parameter-efficient fine-tuning with dynamic rank distribution.arXiv preprint arXiv:2405.17357. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    Pith/arXiv arXiv

  7. [16]

    Xuyin Qi, Zeyu Zhang, Canxuan Gang, Hao Zhang, Lei Zhang, Zhiwei Zhang, and Yang Zhao

    Adapterfusion: Non-destructive task composition for transfer learning.arXiv preprint arXiv:2005.00247. Xuyin Qi, Zeyu Zhang, Canxuan Gang, Hao Zhang, Lei Zhang, Zhiwei Zhang, and Yang Zhao. 2025a. Mediaug: Exploring visual augmentation in medi- cal imaging. InAnnual Conference on Medical Im- age Understanding and Analysis, pages 218–232. Springer. Xuyin Q...

    Pith/arXiv arXiv 2005

  8. [17]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts

    Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts

    Pith/arXiv arXiv

  9. [19]

    Alex Wang

    Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation.arXiv preprint arXiv:2210.07558. Alex Wang

    Pith/arXiv arXiv

  10. [20]

    arXiv preprint arXiv:1804.07461

    Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. F. Wang et al

    Pith/arXiv arXiv

  11. [21]

    Adina Williams, Nikita Nangia, and Samuel R Bow- man

    Neural network acceptability judg- ments.arXiv preprint arXiv:1805.12471. Adina Williams, Nikita Nangia, and Samuel R Bow- man

    Pith/arXiv arXiv

  12. [22]

    Thomas Wolf

    A broad-coverage challenge corpus for sentence understanding through inference.arXiv preprint arXiv:1704.05426. Thomas Wolf

    Pith/arXiv arXiv

  13. [23]

    Yu-Hang Wu, Yu-Jie Xiong, Hao Zhang, Jia-Chen Zhang, and Zheng Zhou

    Transformers: State-of-the- art natural language processing.arXiv preprint arXiv:1910.03771. Yu-Hang Wu, Yu-Jie Xiong, Hao Zhang, Jia-Chen Zhang, and Zheng Zhou

    Pith/arXiv arXiv 1910

  14. [24]

    Xi Xiao, Aristeidis Tsaris, Anika Tabassum, John Lager- gren, Larry M

    Sugar-coated poison: Benign generation unlocks llm jailbreaking.arXiv preprint arXiv:2504.05652. Xi Xiao, Aristeidis Tsaris, Anika Tabassum, John Lager- gren, Larry M. York, Tianyang Wang, and Xiao Wang. 2025a. Focus: Fused observation of channels for un- veiling spectra.Preprint, arXiv:2507.14787. Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao W...

    arXiv

  15. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al

    Pith/arXiv arXiv

  16. [26]

    5 tech- nical report.arXiv preprint arXiv:2412.15115

    Qwen2. 5 tech- nical report.arXiv preprint arXiv:2412.15115. E. Zaken, Y . Goldberg, and S. Ravfogel

    Pith/arXiv arXiv

  17. [27]

    Bitfit: Simple parameter-efficient fine-tuning for transform- ers.Transactions of the Association for Computa- tional Linguistics (TACL), 10:1–16. D. Zhang et al. 2023a. Parameter-efficient fine-tuning methods for llms.Journal of Machine Learning Re- search. Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. 2023b. In- cr...

    Pith/arXiv arXiv 2025

  18. [28]

    Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning.arXiv preprint arXiv:2403.09113. B. Zhu et al. 2023a. Expanding frontiers in large lan- guage models.AI Frontier Research. B. Zhu et al. 2023b. Large language models: Progress and applications.Advances in NLP. A Experimental Setup A.1 Baselines We adopt several repre...

    Pith/arXiv arXiv 2019

  19. [29]

    The details of the approxi- mate Hessian matrix computation can be found in Section 4.6

    and WikiText2 (Mer- ity et al., 2016), we obtain nearly identical results for the Hessian matrix, which further validates the stability of our method. The details of the approxi- mate Hessian matrix computation can be found in Section 4.6. We designate the local metric ST opk with k set to half of the total number of diagonal elements, and set the paramet...

    2016

  20. [2013]

    InProceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631–1642

    Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631–1642. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi

    2013

  21. [2016]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu

    Pointer sentinel mixture mod- els.Preprint, arXiv:1609.07843. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu

    Pith/arXiv arXiv

  22. [2017]

    Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.arXiv preprint arXiv:1708.00055. G. Ding et al

    Pith/arXiv arXiv 2017

  23. [2018]

    Measuring the intrinsic di- mension of objective landscapes.arXiv preprint arXiv:1804.08838. X. Li and P. Liang

    Pith/arXiv arXiv

  24. [2019]

    Roberta: A robustly opti- mized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Pith/arXiv arXiv 1907

  25. [2020]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al

    Pith/arXiv arXiv 2012

  26. [2021]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021:3045–3061

    The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021:3045–3061. C. Li et al

    2021

  27. [2022]

    arXiv preprint arXiv:2210.17323

    Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al

    Pith/arXiv arXiv

  28. [2023]

    Platypus: Quick, cheap, and powerful refinement of llms.arXiv preprint arXiv:2308.07317. H.Y . Leong and Y . Wu

    Pith/arXiv arXiv

  29. [2024]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly

    The llama 3 herd of mod- els.arXiv preprint arXiv:2407.21783. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly

    Pith/arXiv arXiv

  30. [2025]

    Chin-Yew Lin

    Gptqv2: Efficient finetuning-free quantization for asymmetric calibra- tion.arXiv preprint arXiv:2504.02692. Chin-Yew Lin

    Pith/arXiv arXiv