REVIEW 10 cited by
Overtrained Language Models Are Harder to Fine-Tune
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.
Forward citations
Cited by 10 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Understanding Reasoning from Pretraining to Post-Training
A joint scaling law: post-RL chess and math performance is predictable from pretraining loss, RL improvement rate grows with pretraining tokens, and RL both amplifies and discovers moves depending on difficulty.
-
Disentangling Geometry, Performance, and Training in Language Models
Effective rank of the unembedding matrix mainly reflects hyperparameters like batch size and weight decay and is not a reliable predictor of language-model performance.
-
The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
A single-weight perturbation at the very start of training makes otherwise identical neural networks diverge to different loss basins, and this sensitivity drops sharply within the first fraction of training.
-
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Applying mHC as a PEFT method shows that learned residual mixing is unnecessary — even harmful — in finetuning, and mHC+LoRA combinations give small task-dependent gains.
-
Reinitializing weights vs units for maintaining plasticity in neural networks
Selective weight reinitialization, which resets the least useful weights, maintains plasticity in small and layer-normalized networks where unit-level reinitialization methods fail.
-
Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
Training-free amplification of selected last-layer activations, combined with 'wait' token insertion, elicits long chain-of-thought reasoning in base LLMs and improves accuracy on math and science benchmarks.
-
The wall confronting large language models
LLM scaling exponents near 0.1 imply that reducing loss tenfold would need 10^10 more compute, making scientific-grade reliability unreachable by brute-force scaling.
Discussion (0). Sign in to comment.