{"id":"61d3b60f-bf04-4c09-b509-039873bbed67","arxiv_id":"2605.29317","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"FoRA combines single-pass diagonal Fisher layer selection with Stiefel-manifold training of LoRA down-projections to outperform LoRA at half the parameter budget and approach AdaLoRA at one-quarter the budget across LLaMA-family models.","lead":"FoRA selects task-informative layers with a cheap Fisher score and trains selected LoRA adapters under a Stiefel orthonormality constraint to cut trainable parameters. If the reported gains hold, it offers a practical route to adapt large models with substantially lower memory and compute than standard LoRA variants.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Single-pass diagonal Fisher layer selection's reliability for identifying task-informative layers is the load-bearing assumption.","rationale":"The reader's weakest assumption matches the methodological hinge point exactly. Full text would supply implementation details (Fisher formula, Stiefel optimizer, exact layer counts), but does not remove the need for validation that the cheap selection actually drives the gains rather than merely correlating with them.","tokens_in":1702,"tokens_out":319,"duration_ms":19170,"concrete_test":"On LLaMA-7B with one GLUE task, recompute layer rankings using (a) the paper's single-pass diagonal Fisher and (b) a 5-epoch accumulated diagonal Fisher or full per-layer gradient norm; retrain FoRA with each ranking at identical budget and compare final accuracy. A >2-point gap indicates the single-pass method is insufficient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claims (outperforming LoRA/DoRA at half budget, near AdaLoRA at quarter budget) rest on Fisher selection of fewer layers rather than rank reduction. This requires the single-pass diagonal Fisher (computed at <1% training cost) to accurately surface layers whose adaptation produces the reported gains. Diagonal approximation discards off-diagonal parameter covariances, and a single forward/backward pass may fail to capture task-specific sensitivity across the full training distribution. If selection is noisy or suboptimal, the super-additive claim (Fisher + Stiefel) cannot be isolated from baseline effects or hyperparameter differences.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces FoRA, a parameter-efficient fine-tuning method that selects task-informative layers via a single-pass diagonal Fisher score (at <1% training cost) and constrains the LoRA down-projection at those layers to the Stiefel manifold to enforce column orthonormality. It claims FoRA outperforms LoRA and DoRA at half their parameter budget while approaching AdaLoRA performance at one-quarter the budget, with consistent results across five LLaMA-family models and twelve backbones from 270M to 32B parameters; the two components are reported to combine super-additively.","tokens_in":1817,"tokens_out":543,"duration_ms":16251,"significance":"If the empirical claims and isolation of the Fisher-plus-Stiefel contribution hold under rigorous controls, the work would be significant for PEFT research by shifting focus from rank reduction to layer selection with cheap Fisher approximation and orthogonal constraints, potentially enabling more parameter-efficient adaptation without sacrificing accuracy.","major_comments":[{"comment":"The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects.","section":"§3 (Method) and §4 (Experiments)"},{"comment":"The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget.","section":"§4 (Experiments) and Table/Figure reporting ablations"},{"comment":"No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims.","section":"§4 (Experiments)"}],"minor_comments":[{"comment":"Typo in abstract: 'receivedcomparatively' should be 'received comparatively'.","section":"Abstract"},{"comment":"Notation for the diagonal Fisher score and Stiefel constraint should be formalized with equations early in the method section for clarity.","section":"§3 (Method)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on validation of the Fisher approximation, ablation clarity, and experimental reporting. We address each major comment below with clarifications from the manuscript and indicate planned revisions where appropriate.","responses":[{"response":"The manuscript motivates the single-pass diagonal Fisher by its negligible cost (<1% of training) and its use in prior PEFT literature for layer importance. While direct head-to-head comparisons against multi-pass or full Fisher are not present, the selection's effectiveness is evidenced by the consistent outperformance over random or uniform layer selection baselines at matched budgets. We will add a limited sensitivity study (on 2-3 tasks) comparing single-pass vs. multi-pass Fisher in the revision to further isolate its contribution.","revision_made":"yes","referee_comment":"[§3 (Method) and §4 (Experiments)] The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects."},{"response":"The abstract already states that 'Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.' These component-wise results appear in the full experimental section with matched-parameter comparisons. To improve clarity, we will expand the ablation tables/figures in the revision to explicitly label and juxtapose Fisher-only, Stiefel-only, and combined variants against LoRA/DoRA at identical budgets.","revision_made":"yes","referee_comment":"[§4 (Experiments) and Table/Figure reporting ablations] The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget."},{"response":"Section 4 of the full manuscript specifies the protocol (hyperparameters, datasets, evaluation), reports mean ± std over three random seeds with error bars in all tables/figures, and includes paired statistical tests for key comparisons. The abstract and summary are necessarily concise; we will add a brief pointer to these details in the revised abstract and ensure all figures/tables reference the seed count and significance testing.","revision_made":"partial","referee_comment":"[§4 (Experiments)] No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims."}],"tokens_in":1436,"tokens_out":604,"duration_ms":20579,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"FoRA combines a single-pass diagonal Fisher score to pick which layers to adapt with a Stiefel manifold constraint to keep the down-projections orthogonal. This setup is meant to reduce the number of adapted layers instead of shrinking the rank, and the abstract says it outperforms LoRA and DoRA at half the parameter count while staying within 0.7-0.8 points of AdaLoRA at one quarter the count.\n\nThe pairing of these two techniques is the main new element. Fisher-based selection and orthogonal constraints have appeared before, but applying them together to cut adapted layers and claiming a super-additive effect is the angle here. The results span five LLaMA backbones plus tests on Qwen3 and Gemma models from 270M to 32B parameters, which shows some effort at breadth.\n\nThe load-bearing part that looks soft is the reliability of that Fisher selection. It uses a diagonal approximation computed in one pass at under 1% of training cost. That means it ignores how parameters covary and may not reflect sensitivity over the whole training set. Without strong ablations that isolate whether the selected layers are truly better than random or other heuristics at the same budget, the gains could be driven by other choices. The abstract mentions consistent numerical gains but the full paper needs to show the experimental controls clearly.\n\nThis kind of work is for researchers and engineers working on parameter-efficient fine-tuning who care about memory savings on large models. A reader who wants concrete recipes for lower-budget adaptation could find the reported tradeoffs useful. The paper has enough scale in its experiments to deserve a serious referee, even with the open questions on the selection method.\n\nI would send it for peer review but ask the authors to add detailed ablations on the Fisher component and statistical significance.","headline":"FoRA pairs cheap Fisher layer selection with Stiefel orthogonality to cut adapted layers rather than rank, but the selection step's reliability is the part that needs checking.","tokens_in":2357,"tokens_out":442,"would_cite":false,"duration_ms":25621,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"FoRA selects task-informative layers via cheap Fisher scores and enforces orthonormality on LoRA down-projections to cut parameters while preserving accuracy.","keywords":["parameter efficient fine tuning","LoRA","Fisher information","Stiefel manifold","large language models","model adaptation","orthogonal constraints"],"falsifier":"Running the method with randomly chosen layers instead of Fisher-selected ones and observing no drop in performance would show that the selection step is not necessary.","tokens_in":2617,"feed_emoji":"📉","tokens_out":623,"duration_ms":23985,"temperature":0.7,"pith_summary":"The paper introduces FoRA to improve parameter efficiency in fine-tuning by focusing on which layers to adapt rather than how much to adapt each one. It computes a diagonal Fisher score in a single cheap pass to pick task-informative layers and then optimizes the LoRA down-projection on the Stiefel manifold to keep columns orthonormal. This combination lets it beat LoRA and DoRA with half the trainable parameters and stay close to AdaLoRA with only a quarter, on multiple large language model families.","feed_headline":"Fisher selection and Stiefel training halve LoRA's parameter needs","feed_subtitle":"Selecting informative layers cheaply and enforcing orthonormality on projections beats standard LoRA at half budget across model sizes.","key_machinery":"Fisher-based layer selection combined with Stiefel manifold training of the LoRA down-projection to enforce orthonormality.","core_discovery":"FoRA selects task-informative layers with a single-pass diagonal Fisher score computed at under 1% training cost and trains the LoRA down-projection matrices at those layers on the Stiefel manifold to preserve column orthonormality and effective rank, resulting in consistent outperformance of LoRA and DoRA at half the parameter budget and near parity with AdaLoRA at one-quarter the count across five LLaMA-family backbones and twelve models from LLaMA, Qwen3, and Gemma families.","pith_inferences":["The layer selection strategy could be applied to other adaptation methods like full fine-tuning or other adapters.","The low-cost Fisher approximation might enable dynamic layer selection during training.","This suggests that many layers in LLMs are less critical for specific tasks, pointing to potential for more sparse adaptation."],"forward_implications":["The Fisher selection and Stiefel constraint combine super-additively for the accuracy gains.","Fisher selection alone can match the performance of rank reduction at the same budget.","The method scales consistently from 270M to 32B parameters.","Gains hold across different model families including Qwen3 and Gemma."],"fun_headline_variants":["FoRA halves LoRA parameters with Fisher layer selection and Stiefel training","Fisher selection and Stiefel training halve LoRA parameter budget","Stiefel training on Fisher selected layers cuts LoRA parameters in half","Fisher score selects layers for Stiefel LoRA at half parameter budget"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The diagonal Fisher score from a single forward-backward pass on a small data subset accurately identifies the most task-informative layers for adaptation.","fun_headline_variants_meta":{"raw":{"variants":["FoRA halves LoRA parameters with Fisher layer selection and Stiefel training","Fisher selection and Stiefel training halve LoRA parameter budget","Stiefel training on Fisher selected layers cuts LoRA parameters in half","Fisher score selects layers for Stiefel LoRA at half parameter budget"]},"model":"grok-4.3","cost_usd":0.013444,"raw_usage":{"total_tokens":5823,"prompt_tokens":676,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":134437000,"prompt_tokens_details":{"text_tokens":676,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5071,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":676,"tokens_out":76,"duration_ms":32611,"temperature":1.0,"reasoning_tokens":5071,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:48:15.637109+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the method with randomly chosen layers instead of Fisher-selected ones and observing no drop in performance would show that the selection step is not necessary.","supporting_citations":[],"review_version":1}