Pith. sign in

REVIEW 16 cited by

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.02948 v4 pith:NYYNY7G3 submitted 2024-04-03 cs.LG cs.AI

Fanxu Meng , Zhaohui Wang , Muhan Zhang This is my paper · ORCID
classification cs.LGcs.AI
keywords pissaloramathbbprincipalsingulartimesadaptationcomponents
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes $\Delta W \in \mathbb{R}^{m \times n}$ through the product of two matrices $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$, $A$ is initialized with Gaussian noise, and $B$ with zeros. LoRA freezes the original model $W$ and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices $A$ and $B$ with the principal components of the original matrix $W$, and put the remaining components into a residual matrix $W^{res} \in \mathbb{R}^{m \times n}$ which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 12 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, surpassing LoRA's 67.7% by 5.16%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05%, exceeding the performances of QLoRA at 81.73%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA. Code is available at https://github.com/GraphPKU/PiSSA.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spectral Rewiring for Exploration, Purification, and Model Merging

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.

  2. \k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Selecting the top 50% of LoRA weight matrices by condition number halves trainable parameters and cuts fine-tuning time by about 16% while roughly matching full-LoRA accuracy.

  3. ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Spectrum-initialized LoRA with elbow ranks and recursive SVD consolidation of the effective weight beats rank-swept PEFT baselines on three of four 7–8B models in continual GLUE fine-tuning.

  4. Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

    cs.LG 2026-01 reject novelty 6.0 of 10

    SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.

  5. ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.

  6. Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Gradually increasing the probability that LoRA adapters stay active during fine-tuning improves generalization, merging, and pruning robustness.

  7. LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

    cs.LG 2025-02 conditional novelty 6.0 of 10

    LowRA enables LoRA fine-tuning with base weights at 1.15 to 4 bits per parameter, outperforming QLoRA and LoftQ at equal bit widths and matching their accuracy at lower bit widths.

  8. HRP: High-Rank Preheating for Superior LoRA Initialization

    cs.LG 2025-02 conditional novelty 6.0 of 10

    HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.

  9. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  10. ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Tuning only 8.7M parameters of a frozen DINOv2 on daytime data is reported to beat prior PEFT, full fine-tuning, synthetic-data depth methods, and Depth Anything V2 on zero-shot adverse-weather benchmarks.

  11. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  12. Weight Spectra Induced Efficient Model Adaptation

    cs.LG 2025-05 reject novelty 4.0 of 10

    Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.

  13. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

  14. CoLA: Collaborative Low-Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.

  15. Dual Decomposition of Weights and Singular Value Low Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DuDe combines DoRA's magnitude-direction decomposition with PiSSA's SVD-based initialization, reporting consistent but modest accuracy gains over LoRA, DoRA, and PiSSA on commonsense reasoning, GPQA, MMLU, and GSM8K.

  16. PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint

    cs.LG 2025-09 conditional novelty 3.0 of 10

    PHLoRA extracts LoRA-compatible adapters from full-rank fine-tuned models via truncated SVD of the weight delta, matching full-rank performance on several benchmarks with no gradients or training data.

Pith tools