pith. sign in

arxiv: 2607.01232 · v2 · pith:LJ364HT3new · submitted 2026-07-01 · 💻 cs.LG · cs.CL

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Zijian Zhang , Rizhen Hu , Athanasios Glentis , Dawei Li , Chung-Yiu Yau , Hongzhou Lin , Mingyi Hong This is my paper

Pith reviewed 2026-07-03 21:29 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords reinforcement learningtransformer layersLLM post-traininglayer-wise analysisGRPOmodel adaptationmiddle layersRL efficiency
0
0 comments X

The pith

Training a single middle transformer layer recovers most gains from full-parameter RL post-training on LLMs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper challenges the assumption that all transformer layers contribute equally during RL adaptation of LLMs. Experiments across seven models, three RL algorithms, and tasks like math reasoning and code generation show that performance gains concentrate in a small subset of layers, frequently just one in the middle of the stack. Training only that layer in isolation recovers most or all of the improvement from updating every parameter, and sometimes exceeds it. The same middle-heavy pattern appears consistently regardless of model family, dataset, or algorithm. This indicates that uniform full-parameter updates are often unnecessary for RL post-training.

Core claim

By introducing a layer contribution metric that measures the fraction of full RL improvement recovered when each layer is trained in isolation, the authors establish that RL gains are highly concentrated in middle layers of the transformer stack. Across Qwen3 and Qwen2.5 models using GRPO, GiGPO, and Dr. GRPO on mathematical, code, and agentic tasks, a single high-contribution layer frequently matches or surpasses the results of full-parameter training, while input and output layers contribute far less.

What carries the argument

The layer contribution metric, which quantifies the fraction of full RL improvement recovered by training one layer in isolation while freezing the rest.

If this is right

  • RL post-training can be performed by updating only a few high-contribution layers while freezing the rest.
  • Middle layers handle the core adaptations needed for reasoning and decision-making under RL.
  • Layer contribution rankings stay stable across tasks and algorithms, allowing reuse of identified layers.
  • Uniform updates waste compute on low-contribution layers near the input and output ends.
  • Targeted layer training reduces the cost of RL adaptation for larger models without losing most gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • High-contribution layers could be the focus of targeted model editing methods for task-specific RL adaptations.
  • The middle-layer concentration may reflect how transformers integrate features for complex outputs.
  • Freezing low-contribution layers during RL could preserve performance while lowering memory use.
  • Similar concentration patterns might appear in supervised fine-tuning or other post-training methods.

Load-bearing premise

Performance when training a layer by itself accurately reflects that layer's role when all layers update together, without important interactions that only arise in joint training.

What would settle it

Retraining the full model while updating only the layer previously identified as high-contribution in isolation and checking whether the resulting performance gain falls substantially short of the isolated-layer result.

Figures

Figures reproduced from arXiv: 2607.01232 by Athanasios Glentis, Chung-Yiu Yau, Dawei Li, Hongzhou Lin, Mingyi Hong, Rizhen Hu, Zijian Zhang.

Figure 1
Figure 1. Figure 1: (a) Layer contribution (defined in §2.2) across all seven models studied in this work, plotted against depth [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Layer contribution C(k) across model scales. Blue: math contribution (in-domain). Black: overall contribution (averaged across all capabilities). Dashed line indicates full-parameter training (C = 1.0). Each point represents one layer trained in isolation. Math and overall contribution closely track each other across layers (Pearson r > 0.6 on 1.7B,4B and 8B), indicating that high-contribution layers achie… view at source ↗
Figure 2
Figure 2. Figure 2: Layer contribution C(k) across model scales. Blue: math contribution (in-domain). Black: overall contribution (averaged across all capabilities). Dashed line indicates full-parameter training (C = 1.0). Each point represents one layer trained in isolation. Math and overall contribution closely track each other across layers (Pearson r > 0.6 on 1.7B,4B and 8B), indicating that high-contribution layers achie… view at source ↗
Figure 3
Figure 3. Figure 3: Cross-dataset consistency of layer contribution on Qwen3-1.7B-Base. Each point represents a single layer. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer contribution C(k) for Qwen2.5-Math-1.5B (28 layers) trained with Dr. GRPO. Each point corresponds to one transformer layer trained in isolation. The dashed line marks full-parameter training (C = 1.0); circled markers indicate layers that reach or exceed it. Despite the change in both model family and RL algorithm, the contribution profile retains the same structure observed on Qwen3: middle layers c… view at source ↗
Figure 5
Figure 5. Figure 5: Layer contribution C(k) on the agentic task ALFWorld, trained with GiGPO. (a) Qwen2.5-1.5B-Instruct (28 layers). (b) Qwen2.5-3B-Instruct (36 layers). A representative subset of layers is trained due to computational constraints. The dashed line marks full-parameter training (C = 1.0); circled markers indicate layers that reach or exceed it. Despite the shift from mathematical reasoning to multi-step agenti… view at source ↗
Figure 6
Figure 6. Figure 6: Layer contribution C(k) for DeepSeek-Distilled-Qwen-7B (28 layers) trained with GRPO on the Skywork mathematics dataset. Only a subset of layers (0, 4, 8, 12, 14, 16, 20, 24) are trained due to computational constraints. The dashed line marks full-parameter training (C = 1.0); the circled marker indicates the layer that exceeds it. Despite differing from the Qwen3 and Qwen2.5 models in both pretraining rec… view at source ↗
Figure 7
Figure 7. Figure 7: Layer contribution-guided training strategies across model scales. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Majority voting results on OlympiadBench (Qwen3-1.7B-Base). Voting across [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Per-layer weight change magnitude ∥∆θk∥2 on Qwen3-1.7B-Base. Blue: full-parameter training (all layers change). Colored spikes: single-layer training (only the trained layer changes; all others remain at zero). Under full training, the weight change is relatively uniform across layers, contrasting with the highly non-uniform layer contribution profile. Under single-layer training, all trained layers underg… view at source ↗
read the original abstract

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript claims that in RL post-training of LLMs, performance gains are highly concentrated in a small subset of (often a single middle) transformer layers. Training only that layer in isolation recovers most or all of the gains from full-parameter RL updates, with the same middle-layer concentration pattern holding stably across seven models (Qwen3/Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and tasks in math, code, and agentic domains. The authors introduce a 'layer contribution' metric based on isolated-layer training to quantify this.

Significance. If the isolation-based metric is shown to be a faithful proxy for marginal contributions under joint optimization, the result would be significant for both practical RL efficiency (targeted layer updates) and theoretical understanding of how RL adaptations distribute across transformer stacks. The cross-model/algorithm/task consistency is a strength of the empirical design.

major comments (2)
  1. [Layer contribution definition and experimental protocol] The central claim rests on the layer contribution metric obtained by training each layer in isolation (others frozen). This implicitly assumes additive, independent contributions, yet residual streams, attention mixing, and the shared RL objective create non-additive dynamics: an update to layer L alters the gradient landscape for layer M. No ablation (e.g., joint training of the top-k layers or gradient-masking controls) is reported to test whether isolated performance correlates with marginal value under full-parameter training.
  2. [Experiments and results sections] The abstract and results report stable layer rankings but supply no information on experimental controls, number of random seeds, statistical significance testing, or potential confounds such as learning-rate rescaling or optimizer adjustments when comparing single-layer vs. full-parameter runs. These details are load-bearing for the claim that single-layer training 'matches or surpasses' full training.
minor comments (1)
  1. [Methods] Notation for the layer contribution quantity should be defined explicitly with an equation rather than described only in prose.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comments point by point below.

read point-by-point responses

  1. Referee: The central claim rests on the layer contribution metric obtained by training each layer in isolation (others frozen). This implicitly assumes additive, independent contributions, yet residual streams, attention mixing, and the shared RL objective create non-additive dynamics: an update to layer L alters the gradient landscape for layer M. No ablation (e.g., joint training of the top-k layers or gradient-masking controls) is reported to test whether isolated performance correlates with marginal value under full-parameter training.

    Authors: The layer contribution is defined as the fraction of the full RL improvement that is recovered by training only that layer in isolation. While we acknowledge the potential for non-additive interactions in the full joint optimization, the metric provides a direct and reproducible way to quantify the performance impact of each layer. The stability of the middle-layer concentration pattern across diverse models, algorithms, and tasks suggests that the isolated training captures a genuine aspect of where RL adaptations are most effective. We will add ablations involving joint training of the highest-contributing layers to further investigate the correlation with marginal contributions under full training. revision: partial

  2. Referee: The abstract and results report stable layer rankings but supply no information on experimental controls, number of random seeds, statistical significance testing, or potential confounds such as learning-rate rescaling or optimizer adjustments when comparing single-layer vs. full-parameter runs. These details are load-bearing for the claim that single-layer training 'matches or surpasses' full training.

    Authors: We will revise the manuscript to include detailed information on the experimental setup, including the number of random seeds, statistical significance testing procedures, and any adjustments made to learning rates or optimizers for the single-layer experiments compared to full-parameter training. revision: yes

Circularity Check

0 steps flagged

No circularity: claims rest on direct empirical isolation measurements

full rationale

The paper defines layer contribution explicitly as an empirical ratio (fraction of full RL gain recovered when training one layer with others frozen) and reports measured values across models, algorithms, and tasks. No derivation chain, equations, or fitted parameters are presented that reduce the central claim to its own inputs by construction. No self-citations are invoked to justify uniqueness or load-bearing premises, and the reported patterns are observational outcomes rather than self-referential predictions. The study is therefore self-contained against its own experimental benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; central claim rests on empirical layer-wise measurements. No free parameters, axioms, or invented entities are described.

pith-pipeline@v0.9.1-grok · 5800 in / 1126 out tokens · 35534 ms · 2026-07-03T21:29:33.096598+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 6 internal anchors

  1. [1]

    Group-in-Group Policy Optimization for LLM Agent Training

    URLhttps://arxiv.org/abs/2505.10978. Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    URL https://arxiv.org/abs/2603.12228. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ru...

    work page arXiv

  3. [3]

    doi: 10.1038/s41586-025-09422-z

    ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URLhttp://dx.doi.org/10.1038/s41586-025-09422-z. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report,

    work page doi:10.1038/s41586-025-09422-z

  4. [4]

    Skywork Open Reasoner 1 Technical Report

    URLhttps://arxiv.org/abs/2505.22312. Anshul Kumar, Gagan Raj Gupta, and Manisha Chawla. Adagradselect: An adaptive gradient-guided layer selection method for efficient fine-tuning of slms,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    URLhttps://arxiv.org/abs/2512.15764. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath-CoT](https://github.com/ project-numina/ai...

    work page arXiv

  6. [6]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin

    URLhttps://arxiv.org/abs/2511.00056. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. InSecond Conference on Language Modeling,

    work page arXiv

  7. [7]

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang

    URL https://arxiv.org/abs/2506.22638. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning,

    work page arXiv

  8. [8]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    URLhttps://arxiv.org/abs/2402.03300. Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang, Yujie Feng, and Xiao-Ming Wu. Under- standing layer significance in llm alignment,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht

    URLhttps://arxiv.org/abs/2410.17875. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning,

    work page arXiv

  10. [10]

    URL https://arxiv.org/abs/2010. 03768. Xinyuan Song, Keyu Wang, PengXiang Li, Lu Yin, and Shiwei Liu. Demystifying the roles of llm layers in retrieval, knowledge, and reasoning,

    work page 2010

  11. [11]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    URLhttps://arxiv.org/abs/2510.02091. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models,

    work page arXiv

  12. [12]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    URL https://arxiv. org/abs/2203.11171. 16 Is One Layer Enough? Training a Single Transformer Layer Can Match Full-Parameter RL Training An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    URL https://arxiv.org/abs/2505.09388. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xi...

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    URL https: //arxiv.org/abs/2503.14476. Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Appendix A Training Details and Hyperparameters A.1 Overview Table 1 in the main text summarizes all seven models and their training configurations

    URL https://arxiv.org/abs/2409.14381. Appendix A Training Details and Hyperparameters A.1 Overview Table 1 in the main text summarizes all seven models and their training configurations. Below we provide full hyperparameter details for each experimental setup. In all cases, single-layer training freezes every parameter except the target decoder layer (inc...

    work page arXiv 2025