FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning

Han-Sang Lee; Jaeho Lee; Juneyoung Park; Kiduk Kwon; Kyuho Lee; Minjae Kim; Seongbae Lee; Seongwan Kim; Seungheon Hyeon

arxiv: 2605.29317 · v2 · pith:55COIABHnew · submitted 2026-05-28 · 💻 cs.CL

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning

Juneyoung Park , Seongbae Lee , Han-Sang Lee , Kyuho Lee , Minjae Kim , Seungheon Hyeon , Kiduk Kwon , Seongwan Kim

show 1 more author

Jaeho Lee

This is my paper

Pith reviewed 2026-06-29 07:48 UTC · model grok-4.3

classification 💻 cs.CL

keywords parameter efficient fine tuningLoRAFisher informationStiefel manifoldlarge language modelsmodel adaptationorthogonal constraints

0 comments

The pith

FoRA selects task-informative layers via cheap Fisher scores and enforces orthonormality on LoRA down-projections to cut parameters while preserving accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FoRA to improve parameter efficiency in fine-tuning by focusing on which layers to adapt rather than how much to adapt each one. It computes a diagonal Fisher score in a single cheap pass to pick task-informative layers and then optimizes the LoRA down-projection on the Stiefel manifold to keep columns orthonormal. This combination lets it beat LoRA and DoRA with half the trainable parameters and stay close to AdaLoRA with only a quarter, on multiple large language model families.

Core claim

FoRA selects task-informative layers with a single-pass diagonal Fisher score computed at under 1% training cost and trains the LoRA down-projection matrices at those layers on the Stiefel manifold to preserve column orthonormality and effective rank, resulting in consistent outperformance of LoRA and DoRA at half the parameter budget and near parity with AdaLoRA at one-quarter the count across five LLaMA-family backbones and twelve models from LLaMA, Qwen3, and Gemma families.

What carries the argument

Fisher-based layer selection combined with Stiefel manifold training of the LoRA down-projection to enforce orthonormality.

If this is right

The Fisher selection and Stiefel constraint combine super-additively for the accuracy gains.
Fisher selection alone can match the performance of rank reduction at the same budget.
The method scales consistently from 270M to 32B parameters.
Gains hold across different model families including Qwen3 and Gemma.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The layer selection strategy could be applied to other adaptation methods like full fine-tuning or other adapters.
The low-cost Fisher approximation might enable dynamic layer selection during training.
This suggests that many layers in LLMs are less critical for specific tasks, pointing to potential for more sparse adaptation.

Load-bearing premise

The diagonal Fisher score from a single forward-backward pass on a small data subset accurately identifies the most task-informative layers for adaptation.

What would settle it

Running the method with randomly chosen layers instead of Fisher-selected ones and observing no drop in performance would show that the selection step is not necessary.

Figures

Figures reproduced from arXiv: 2605.29317 by Han-Sang Lee, Jaeho Lee, Juneyoung Park, Kiduk Kwon, Kyuho Lee, Minjae Kim, Seongbae Lee, Seongwan Kim, Seungheon Hyeon.

**Figure 1.** Figure 1: Overview of FoRA. FoRA uses calibration data to score Transformer layers with diagonal Fisher information, selects the Top-K informative layers for adaptation, and applies Stiefel-constrained LoRA only to those layers while keeping the rest frozen. et al., 2024; Hayou et al., 2024). We combine these two ideas into FoRA (Fisherorthogonal Rank Adaptation): Fisher decides where to adapt, and the Stiefel cons… view at source ↗

**Figure 2.** Figure 2: Sensitivity to layer budget K on LLaMA-3.2- 3B with ntrain = 4096, three seeds. FoRA is essentially flat across K ∈ {7, 11, 14, 21, 28}. other: without a structural constraint, where parameters go and how the rank is cut are roughly interchangeable. The decisive gain comes from the Stiefel constraint: adding it to Fisher-top-K (FoRA) drops PPL by an additional 0.35, well outside the seed-noise band. Laye… view at source ↗

**Figure 3.** Figure 3: Fisher-selected top-10 layer indices for seven [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Sensitivity to layer budget K on Gemma-3-1B-pt (26 layers), evaluated by WikiText-2 PPL. K ∈ {7, 10, 13, 20, 26}. FoRA remains flat across all K values; FG-LoRA degrades as K increases beyond the Fisherselected core. Mean ± half-range over two seeds. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

**Figure 5.** Figure 5: Singular spectrum of [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

read the original abstract

Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

FoRA pairs cheap Fisher layer selection with Stiefel orthogonality to cut adapted layers rather than rank, but the selection step's reliability is the part that needs checking.

read the letter

FoRA combines a single-pass diagonal Fisher score to pick which layers to adapt with a Stiefel manifold constraint to keep the down-projections orthogonal. This setup is meant to reduce the number of adapted layers instead of shrinking the rank, and the abstract says it outperforms LoRA and DoRA at half the parameter count while staying within 0.7-0.8 points of AdaLoRA at one quarter the count.

The pairing of these two techniques is the main new element. Fisher-based selection and orthogonal constraints have appeared before, but applying them together to cut adapted layers and claiming a super-additive effect is the angle here. The results span five LLaMA backbones plus tests on Qwen3 and Gemma models from 270M to 32B parameters, which shows some effort at breadth.

The load-bearing part that looks soft is the reliability of that Fisher selection. It uses a diagonal approximation computed in one pass at under 1% of training cost. That means it ignores how parameters covary and may not reflect sensitivity over the whole training set. Without strong ablations that isolate whether the selected layers are truly better than random or other heuristics at the same budget, the gains could be driven by other choices. The abstract mentions consistent numerical gains but the full paper needs to show the experimental controls clearly.

This kind of work is for researchers and engineers working on parameter-efficient fine-tuning who care about memory savings on large models. A reader who wants concrete recipes for lower-budget adaptation could find the reported tradeoffs useful. The paper has enough scale in its experiments to deserve a serious referee, even with the open questions on the selection method.

I would send it for peer review but ask the authors to add detailed ablations on the Fisher component and statistical significance.

Referee Report

3 major / 2 minor

Summary. The paper introduces FoRA, a parameter-efficient fine-tuning method that selects task-informative layers via a single-pass diagonal Fisher score (at <1% training cost) and constrains the LoRA down-projection at those layers to the Stiefel manifold to enforce column orthonormality. It claims FoRA outperforms LoRA and DoRA at half their parameter budget while approaching AdaLoRA performance at one-quarter the budget, with consistent results across five LLaMA-family models and twelve backbones from 270M to 32B parameters; the two components are reported to combine super-additively.

Significance. If the empirical claims and isolation of the Fisher-plus-Stiefel contribution hold under rigorous controls, the work would be significant for PEFT research by shifting focus from rank reduction to layer selection with cheap Fisher approximation and orthogonal constraints, potentially enabling more parameter-efficient adaptation without sacrificing accuracy.

major comments (3)

[§3 (Method) and §4 (Experiments)] The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects.
[§4 (Experiments) and Table/Figure reporting ablations] The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget.
[§4 (Experiments)] No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims.

minor comments (2)

[Abstract] Typo in abstract: 'receivedcomparatively' should be 'received comparatively'.
[§3 (Method)] Notation for the diagonal Fisher score and Stiefel constraint should be formalized with equations early in the method section for clarity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on validation of the Fisher approximation, ablation clarity, and experimental reporting. We address each major comment below with clarifications from the manuscript and indicate planned revisions where appropriate.

read point-by-point responses

Referee: [§3 (Method) and §4 (Experiments)] The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects.

Authors: The manuscript motivates the single-pass diagonal Fisher by its negligible cost (<1% of training) and its use in prior PEFT literature for layer importance. While direct head-to-head comparisons against multi-pass or full Fisher are not present, the selection's effectiveness is evidenced by the consistent outperformance over random or uniform layer selection baselines at matched budgets. We will add a limited sensitivity study (on 2-3 tasks) comparing single-pass vs. multi-pass Fisher in the revision to further isolate its contribution. revision: yes
Referee: [§4 (Experiments) and Table/Figure reporting ablations] The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget.

Authors: The abstract already states that 'Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.' These component-wise results appear in the full experimental section with matched-parameter comparisons. To improve clarity, we will expand the ablation tables/figures in the revision to explicitly label and juxtapose Fisher-only, Stiefel-only, and combined variants against LoRA/DoRA at identical budgets. revision: yes
Referee: [§4 (Experiments)] No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims.

Authors: Section 4 of the full manuscript specifies the protocol (hyperparameters, datasets, evaluation), reports mean ± std over three random seeds with error bars in all tables/figures, and includes paired statistical tests for key comparisons. The abstract and summary are necessarily concise; we will add a brief pointer to these details in the revised abstract and ensure all figures/tables reference the seed count and significance testing. revision: partial

Circularity Check

0 steps flagged

No circularity; empirical claims rest on external benchmarks

full rationale

The paper presents FoRA as an empirical PEFT method combining diagonal Fisher layer selection with a Stiefel manifold constraint on selected LoRA adapters. All performance claims (outperforming LoRA/DoRA at half budget, near AdaLoRA at quarter budget) are supported by direct comparisons on LLaMA-family and other backbones, not by any internal derivation, equation, or fitted quantity that reduces to itself by construction. No self-citations, uniqueness theorems, or ansatzes are invoked as load-bearing steps. The single-pass Fisher heuristic is presented as a practical choice under 1% training cost, without claiming it is derived from or equivalent to the accuracy results.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract supplies only high-level assumptions; the two load-bearing premises are the reliability of the cheap Fisher score for layer selection and the rank-preserving effect of the Stiefel constraint. No explicit free parameters or new entities are named.

axioms (2)

domain assumption A single-pass diagonal Fisher score identifies task-informative layers at under 1% of training cost
Used to select which layers receive the adapter
domain assumption Training the LoRA down-projection on the Stiefel manifold preserves column orthonormality and effective rank
Stated as the mechanism that maintains performance at low parameter count

pith-pipeline@v0.9.1-grok · 5758 in / 1503 out tokens · 16901 ms · 2026-06-29T07:48:15.637109+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

LoRA learns less and forgets less.TMLR. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. InAAAI. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural ...

work page internal anchor Pith review Pith/arXiv arXiv 2020
[2]

InNeurIPS

Limitations of the empirical Fisher approxima- tion for natural gradient descent. InNeurIPS. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight- decomposed low-rank adaptation. InICML. Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zh...

work page arXiv 2015

[1] [1]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

LoRA learns less and forgets less.TMLR. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. InAAAI. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural ...

work page internal anchor Pith review Pith/arXiv arXiv 2020

[2] [2]

InNeurIPS

Limitations of the empirical Fisher approxima- tion for natural gradient descent. InNeurIPS. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight- decomposed low-rank adaptation. InICML. Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zh...

work page arXiv 2015