REVIEW 20 cited by
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90\% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.
Forward citations
Cited by 20 Pith papers
-
Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation
A forward-only controller sets multi-domain LoRA participation from label-free competence and cross-domain affinity, improving average accuracy while using half the data.
-
Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
Frozen random CNN feature extractors in PPO yield sparse readouts whose active-neuron count is claimed to track task complexity, but the flagship deterministic-Pong numbers are contradicted by the paper's own appendices.
-
The 3D Mirage: Probing and Taming 3D Hallucinations
Depth models hallucinate 3D bumps on flat illusion images when context is cropped; the paper adds a benchmark, two scores, and a LoRA fine-tune that reduces the artifact on the same dataset.
-
Towards Understanding the Shape of Representations in Protein Language Models
Protein language models encode 3D structure best at very local residue distances (around 2 to 8 neighbors) and in layers just before the last, not in the final layer.
-
Implementing Adaptations for Vision AutoRegressive Model
Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.
-
Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
FedASK uses double sketching and server-side SVD to aggregate federated LoRA updates under differential privacy, allowing both A and B matrices to be updated instead of freezing one.
-
ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.
-
Time Series Foundation Models for Multivariate Financial Time Series Forecasting
Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...
-
DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models
A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.
-
Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence
CorDA++ uses data-driven SVD to initialize LoRA adapters, adding per-layer covariance selection and rank allocation that reduce forgetting and speed convergence compared to LoRA, PiSSA, QLoRA, and other baselines.
-
Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification
TensorGuard classifies fine-tuned LLMs into their base-model families with 94% accuracy by clustering statistical features of weight gradients under random input perturbations.
-
ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation
ReqBrain, a LoRA-fine-tuned Zephyr-7b-beta model, produces software requirements that human evaluators could not reliably tell apart from human-authored ones, with automatic metrics favoring it over untuned ChatGPT-4o.
-
Backbone Augmented Training for Adaptations
Selecting backbone pre-training data with an influence-based score and adding it to the adaptation set can improve LoRA and DreamBooth training when adaptation data are scarce.
-
Weight Spectra Induced Efficient Model Adaptation
Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.
-
MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.
-
Generalizing vision-language models to novel domains: A comprehensive survey
A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
-
Slimming Down LLMs Without Losing Their Minds
QLoRA fine-tuning on Alpaca data drops a 1B model's math accuracy from 33.5% to 3.7% and reduces MMLU-CS accuracy by 13 points, while HellaSwag stays flat.
Discussion (0). Sign in to comment.