REVIEW 3 cited by
CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.
T0 review reviewed 2026-08-04 challenge →
CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
CLPO changes what is trained on, step by step. Each training step, the model solves each sampled problem four times. The fraction correct becomes a difficulty score. Problems solved once in four tries are called hard; problems solved twice are called medium. Hard problems are rewritten with a prompt that asks for a simpler version. Medium problems are rewritten with a prompt that asks for a different wording of the same problem. Rewritten problems that the model now solves zero or four times are discarded. The surviving original and rewritten problems are used for the policy update, with a weaker KL penalty on hard problems so the policy can change more there. This loop is fully self-generated, with no external teacher or annotation beyond the original answers.
On Qwen3-8B trained on the DAPO-Math-17k set, CLPO reports an average pass@1 of 73.73 on eight benchmarks, versus 64.50 for DAPO and 62.04 for GRPO, with the largest gains on out-of-domain sets. The core ideas are plausible, but the numbers need scrutiny: the abstract and the table disagree on the size of the gains, all results are single runs with no error bars, the method's hyperparameters were tuned on one of the headline benchmarks, and the model's rewrites are never checked to be mathematically equivalent to the original problems.
Core claim
The central claim is that CLPO 'achieves state-of-the-art performance across eight challenging mathematical and general reasoning benchmarks, with an average pass@1 improvement of 6.96% over other methods' (body abstract), and per the submitted abstract 'substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points.' If correct, the mechanism (online accuracy-based partitioning into hard and medium sets, simplification of hard problems, diversification of medium problems, value-driven filtering, and difficulty-aware KL scaling) yields large average benchmark gains over GRPO and DAPO on Qwen3-8B, including out-of-domain generalization without external guidance.
Load-bearing premise
The load-bearing premise is that the model's self-generated rewrites preserve the original problem's mathematical content and answer, so training on (q', a) with the preserved answer a teaches transferable reasoning. Section 3.4 asserts 'the original answer a is strictly preserved' with no verification step; the value filter in Eq. (8) retains rewrites whose rollout accuracy is strictly between 0 and 1, which does not test equivalence (a degenerate or non-equivalent rewrite can still be partially solvable). If Qwen3-8B, acting as its own teacher, systematically produces rewrites that are not equivalent to the originals, the curriculum signal is corrupted and the reported gains, especially the out-of-domain improvements, would not be attributable to the stated mechanism. A secondary fragile premise is the assumption that the public in-domain eval sets are not contaminated by DAPO-Math-17k training data, which the paper does not analyze.
Editorial analysis
A structured set of objections, weighed in public.
Axiom & Free-Parameter Ledger
free parameters (5)
- hard accuracy threshold tau_hard =
0.3
- medium accuracy threshold tau_med =
0.7
- hard KL scaling factor lambda_hard =
0.3
- non-hard KL scaling factor lambda_non_hard =
1.0
- restructuring prompt templates (diversification and simplification) =
two prompt texts in Appendix D (Figures 7-8)
axioms (5)
- ad hoc to paper Rewrite equivalence: f_p(q, d) preserves the mathematical content and answer a of the original problem q.
- domain assumption Rollout accuracy with G=4 samples at temperature 1.0 is a reliable difficulty signal.
- domain assumption No contaminating overlap between DAPO-Math-17k training data and the in-domain evaluation sets.
- standard math Standard GRPO clipped surrogate objective and KL penalty assumptions (Eqs. (1) and (2)).
- domain assumption The verifier correctly judges correctness on original and rewritten problems.
Cite this review
Pith. "Pith review of CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning." pith.science (2026). https://pith.science/paper/HGXRU3A3
@misc{pith2026250925004,
author = {Pith},
title = {Pith review of: CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/HGXRU3A3}},
note = {Machine review of arXiv:2509.25004}
}
read the original abstract
Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems. We propose \textbf{CLPO (Curriculum Learning meets Policy Optimization)}, a self-evolving curriculum framework that uses on-policy rollout accuracy to identify solved, medium-difficulty, and hard problems, then restructures selected tasks according to the model's current capability. Hard problems are simplified to become learnable, while medium-difficulty problems are diversified to provide useful training variation. This allows the learning curriculum to co-evolve with the policy rather than remaining fixed as the model's capability boundary shifts. Rather than treating these rewrites as static data augmentation, CLPO optimizes restructuring trajectories with credit assigned by the downstream accuracy gain of the rewritten problem, requiring no additional human annotations beyond the original verifiable answers. Experiments across mathematical reasoning and out-of-domain general reasoning benchmarks show that CLPO substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points, respectively. Ablation studies on math and code domains further show that both the restructuring mode and the rewriting loss contribute to the final gains, demonstrating that CLPO provides a scalable and robust pathway for eliciting stronger reasoning capabilities through a self-evolving curriculum.
Figures
Forward citations
Cited by 3 Pith papers
-
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...
-
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.
-
ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
ENVS generates verified supervision via environment-native search in OSWorld VMs to train GUI agents, reaching 30.3 pass@8 on 300 tasks while using less compute than ARPO baselines and introducing OSWorld-Noisy for in...
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[7]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[8]
Hasan Abed Al Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long, think short: Curriculum learning for efficient rea- soning.arXiv preprint arXiv:2508.08940,
-
[9]
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,
-
[11]
Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass@ 1: Self-play with variational problem synthesis sustains rlvr.arXiv preprint arXiv:2508.14029,
-
[12]
V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu
Preprint. V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. InInternational conference on machine learning, pp. 1928–1937. PmLR,
1928
-
[13]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
-
[15]
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning.arXiv preprint arXiv:2504.05520,
-
[16]
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726,
-
[19]
Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems.arXiv preprint arXiv:2505.15201,
-
[20]
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,
-
[21]
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245,
-
[22]
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang
Preprint. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,
-
[23]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[24]
Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
-
[25]
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. Critique-grpo: Advancing llm reasoning with natural language and numerical feedback.arXiv preprint arXiv:2506.03106,
-
[26]
Group sequence policy optimization.arXiv preprint arXiv:2507.18071,
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,
-
[27]
Table 3: CLPO-specific hyperparameters
Software frameworks used. Table 3: CLPO-specific hyperparameters. Name Value Description hard threshold (τhard) 0.3 Accuracy threshold for hard problems. medium threshold (τmed) 0.7 Accuracy threshold for medium problems. hard kl scaler (λhard) 0.3 KL scaling factor for hard problems. non hard kl scaler (λnon-hard) 1.0 KL scaling factor for non-hard probl...
2023
-
[1998]
Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
-
[2009]
Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
1901
-
[2017]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2018]
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient train- ing r1-like reasoning models.arXiv preprint arXiv:2503.17287,
-
[2020]
Theoremqa: A theorem-driven question answering dataset.arXiv preprint arXiv:2305.12524,
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset.arXiv preprint arXiv:2305.12524,
-
[2021]
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,
-
[2023]
Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,
-
[2024]
Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,
-
[2025]
Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.