REVIEW 22 cited by
LoRA Learns Less and Forgets Less
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.
Forward citations
Cited by 22 Pith papers
-
Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5
Emergent misalignment in Qwen2.5 is mediated by a causal persona direction that low-rank LoRA recruits from covert code while full SFT does not and moves against it.
-
Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.
-
MemSFT: Mitigating Alignment Tax with an External Parametric Memory
MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...
-
How Meta-Learning Shapes LoRA Adapter Geometry in Speech Deepfake Detection
Meta-learning training concentrates loss-relevant LoRA updates in query/key projections and spreads them in output projections, relative to standard empirical-risk training.
-
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Routing LoRA adapters with the frozen base router's logits plus a shared cross-layer adapter pool gives the best PEFT accuracy and retention on three MoE backbones.
-
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.
-
After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation
Hyperbolic losses at the loss layer alone train stably at 18M-sample scale, while the same geometry inside a trainable adapter is claimed to collapse training.
-
Pusa V1.0: Unlocking Temporal Control in Pretrained Video Diffusion Models via Vectorized Timestep Adaptation
A $500 LoRA fine-tune of Wan2.1-T2V with per-frame random timesteps matches Wan-I2V's benchmark quality and adds zero-shot start-end and video-extension capabilities.
-
Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving
R2SE refines pretrained end-to-end driving policies on hard cases via residual LoRA reinforcement learning and switches between specialist and generalist policies using GPD-based uncertainty.
-
Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
SEFT fine-tunes sparse LLMs by dynamically growing and pruning a sparse set of weight updates, preserving target sparsity while improving downstream task performance over sparsity-preserving baselines.
-
Scaling Point-in-Time Language Models
Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.
-
One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
Soft-prompt teachers, each a small set of learnable tokens on a frozen backbone, can replace full fine-tuning as on-policy distillation supervisors and compose across tasks into a single student.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.
-
GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay
A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.
-
EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.
-
Improving LoRA with Variational Learning
Replacing AdamW with IVON and pruning 10% of highest-variance LoRA parameters improves average commonsense-reasoning accuracy by 1.3 points and calibration by 5.4 points on Llama-3.2-3B.
-
SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.
-
ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics
Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.
-
LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning
LARGO adds per-matrix learned projection radii and SVD-based initialization to LoRA and reports improved out-of-distribution accuracy on DomainNet, ImageNet, and VQA benchmarks.
-
Weight Spectra Induced Efficient Model Adaptation
Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.
-
Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting
MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight ...
Discussion (0). Sign in to comment.