Pith. sign in

REVIEW 18 cited by

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2002.06305 v1 pith:PDT6RFZR submitted 2020-02-15 cs.CL cs.LG

Jesse Dodge , Gabriel Ilharco , Roy Schwartz , Ali Farhadi , Hannaneh Hajishirzi , Noah Smith This is my paper
classification cs.CLcs.LG
keywords fine-tuningtrainingdataperformancerandomtrialsweightdatasets
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 216 citations worldwide. Full citation record

  1. Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Apparent phase transitions during fine-tuning on near-synonym tasks are phantoms originating in the softmax readout; an order parameter isolates kinematic and structural failure modes and a few dimensionless quantitie...

  2. What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Inference backend choice alone can shift LLM benchmark scores by up to 0.055 mean absolute divergence and change which questions are answered correctly, even under greedy decoding.

  3. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

  4. In-Context Probing for Membership Inference in Fine-Tuned Language Models

    cs.CR 2025-12 conditional novelty 6.0 of 10

    ICP-MIA infers membership in fine-tuned LLMs by measuring confidence improvement under in-context probes, beating prior black-box attacks at low false-positive rates.

  5. LobRA: Multi-tenant Fine-tuning over Heterogeneous Data

    cs.DC 2025-09 accept novelty 6.0 of 10

    LobRA reduces GPU seconds for multi-tenant LoRA fine-tuning by 45.03%-60.67% through heterogeneous FT replicas and per-step workload-balanced dispatching.

  6. Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Representation smoothness can be used to regularize training, stop early without validation labels, and guide active learning combined with parameter-efficient fine-tuning, reducing data and compute.

  7. Can Interpretation Predict Behavior on Unseen Data?

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Presence of hierarchical attention heads on in-distribution data predicts hierarchical out-of-distribution generalization across 270 small transformers, independent of causal support.

  8. Breaking a Logarithmic Barrier in the Stopping Time Convergence Rate of Stochastic First-order Methods

    math.OC 2025-06 conditional novelty 6.0 of 10

    A new concentration lemma for almost super-martingales yields a stopping-time convergence rate of O(log(tau)/sqrt(tau)) for a momentum SGD variant, removing one logarithmic factor.

  9. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  10. Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Frozen-embedding GMMs route compact SVD-subspace LoRA adapters for task-agnostic continual learning with SOTA average performance and near-zero forgetting.

  11. Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

    cs.LG 2025-12 conditional novelty 5.0 of 10

    DTS approximates task-specific model deltas with truncated SVD plus grouped thresholding and scaling, achieving near-individual multi-task accuracy with ~1% extra storage per task.

  12. SindBERT, the Sailor: Charting the Seas of Turkish NLP

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SindBERT releases Turkish RoBERTa base/large models trained on 312GB of text; they match existing models, with the large variant best on two of four tasks and little scaling gain.

  13. Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification

    cs.LG 2025-06 conditional novelty 5.0 of 10

    TensorGuard classifies fine-tuned LLMs into their base-model families with 94% accuracy by clustering statistical features of weight gradients under random input perturbations.

  14. Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation

    cs.SE 2025-06 conditional novelty 5.0 of 10

    Nine LLMs generated 3,373 UML methods across 90 diagrams from waste-management use cases, producing mostly valid, structurally faithful output with notable cross-model naming consensus.

  15. Beyond Transfer Accuracy: Faithful Circuits for Controlled Low-Resource Adaptation

    cs.CL 2026-01 reject novelty 4.0 of 10

    A mechanism-guided fine-tuning method that updates only task-relevant attention heads (found via adapted CD-T) can reduce catastrophic forgetting in low-resource cross-lingual transfer, though the headline results rel...

  16. GeistBERT: Breathing Life into German NLP

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A 126M-parameter German BERT, pretrained further on 1.3TB of mixed German text, beats other base models on most tested German NLP benchmarks.

  17. Revisiting Bayesian Model Averaging in the Era of Foundation Models

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper proposes Bayesian model averaging and an entropy-minimizing weight optimizer for ensembling foundation models, reporting accuracy gains over output averaging on image and text classification tasks.

  18. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools