Pith. sign in

REVIEW 9 cited by

Learning Dynamics of LLM Finetuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.10490 v4 pith:IH7UWDP3 submitted 2024-07-15 cs.LG cs.AIcs.CL

Yi Ren , Danica J. Sutherland This is my paper
classification cs.LGcs.AIcs.CL
keywords learningfinetuningdynamicsframeworkmodeldifferentexamplesmight
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.

  2. Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In a linear model of LLM personalization with shared compute, SFT beats ICL above a coverage-dependent signal-to-noise threshold, congestion can reverse that ranking, and adding SFT never reduces platform profit.

  3. Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Re-distilling a model's own RL-trained policy into 1K SFT samples reproduces RL accuracy at a fraction of the compute.

  4. An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Emergent misalignment and realignment are brittle surface effects driven by dataset artifacts like response length rather than stable representational changes.

  5. Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation

    cs.CV 2026-03 unverdicted novelty 5.0 of 10

    A text-to-3D pipeline that predicts a global semantic-geometric layout, conditions panoramic and video diffusion on it, and fuses views with 3D Gaussian Splatting into an absolute-scale navigable indoor scene.

  6. Decoupling Task-Solving and Output Formatting in LLM Generation

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A decoding-time method that keeps the format in a separate module improves LLM accuracy by 1–6% with guaranteed format compliance on math, judging, and extraction.

  7. RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs

    cs.LG 2025-08 reject novelty 5.0 of 10

    Reinforcement fine-tuning largely recovers out-of-distribution performance lost during supervised fine-tuning by counteracting rotations of singular vector directions, with low-rank or shallow-layer restoration recove...

  8. Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

    cs.LG 2025-06 conditional novelty 5.0 of 10

    SFT is cast as a special case of implicit reward learning, and a slower SFT learning rate improves downstream DPO/SimPO results by up to 25% relative win rate.

  9. On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

    cs.LG 2025-05 conditional novelty 5.0 of 10

    GRPO can suffer from Lazy Likelihood Displacement, where correct responses gain little or lose likelihood, and selectively downweighting penalties on harmful tokens (NTHR) mitigates this and boosts math reasoning performance.

Pith tools