Pith. sign in

REVIEW 3 cited by

CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.

T0 review reviewed 2026-08-04 challenge →

arxiv 2509.25004 v2 pith:HGXRU3A3 submitted 2025-09-29 cs.AI

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning

classification cs.AI
keywords reasoningclpocurriculumlearningproblemspolicyaccuracybecome
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models trained with reinforcement learning and verifiable rewards (RLVR) learn to reason by generating many candidate solutions and receiving a reward only when the final answer is correct. Standard methods sample problems from a fixed set until training ends. That wastes compute on problems the model already solves and gives no signal on problems it never solves correctly.

CLPO changes what is trained on, step by step. Each training step, the model solves each sampled problem four times. The fraction correct becomes a difficulty score. Problems solved once in four tries are called hard; problems solved twice are called medium. Hard problems are rewritten with a prompt that asks for a simpler version. Medium problems are rewritten with a prompt that asks for a different wording of the same problem. Rewritten problems that the model now solves zero or four times are discarded. The surviving original and rewritten problems are used for the policy update, with a weaker KL penalty on hard problems so the policy can change more there. This loop is fully self-generated, with no external teacher or annotation beyond the original answers.

On Qwen3-8B trained on the DAPO-Math-17k set, CLPO reports an average pass@1 of 73.73 on eight benchmarks, versus 64.50 for DAPO and 62.04 for GRPO, with the largest gains on out-of-domain sets. The core ideas are plausible, but the numbers need scrutiny: the abstract and the table disagree on the size of the gains, all results are single runs with no error bars, the method's hyperparameters were tuned on one of the headline benchmarks, and the model's rewrites are never checked to be mathematically equivalent to the original problems.

Core claim

The central claim is that CLPO 'achieves state-of-the-art performance across eight challenging mathematical and general reasoning benchmarks, with an average pass@1 improvement of 6.96% over other methods' (body abstract), and per the submitted abstract 'substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points.' If correct, the mechanism (online accuracy-based partitioning into hard and medium sets, simplification of hard problems, diversification of medium problems, value-driven filtering, and difficulty-aware KL scaling) yields large average benchmark gains over GRPO and DAPO on Qwen3-8B, including out-of-domain generalization without external guidance.

Load-bearing premise

The load-bearing premise is that the model's self-generated rewrites preserve the original problem's mathematical content and answer, so training on (q', a) with the preserved answer a teaches transferable reasoning. Section 3.4 asserts 'the original answer a is strictly preserved' with no verification step; the value filter in Eq. (8) retains rewrites whose rollout accuracy is strictly between 0 and 1, which does not test equivalence (a degenerate or non-equivalent rewrite can still be partially solvable). If Qwen3-8B, acting as its own teacher, systematically produces rewrites that are not equivalent to the originals, the curriculum signal is corrupted and the reported gains, especially the out-of-domain improvements, would not be attributable to the stated mechanism. A secondary fragile premise is the assumption that the public in-domain eval sets are not contaminated by DAPO-Math-17k training data, which the paper does not analyze.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a handful of load-bearing choices: two accuracy thresholds and a KL scaling factor tuned on the headlined benchmark; unvalidated rewrite prompts as the teacher mechanism; the assumption that self-rewrites preserve problem-answer equivalence; the coarse G=4 accuracy grid that collapses difficulty bands; and the assumption of no train/test contamination. All are stated or implicit in the paper, and none is independently verified outside the reported runs.

free parameters (5)
  • hard accuracy threshold tau_hard = 0.3
    Selected via ablations on AIME24 (Figure 3, right panel). With G=4 rollouts it defines the hard set as problems with exactly 1 of 4 correct answers.
  • medium accuracy threshold tau_med = 0.7
    Selected via ablations on AIME24. With G=4 rollouts the medium band (tau_hard < Acc <= tau_med) contains only Acc=0.5, i.e., exactly 2 of 4 correct.
  • hard KL scaling factor lambda_hard = 0.3
    Selected via ablation on AIME24 (Figure 4, alpha sweep); 0.3 reported optimal, 0.5 worst.
  • non-hard KL scaling factor lambda_non_hard = 1.0
    Fixed relative anchor equal to the static KL coefficient in the GRPO objective.
  • restructuring prompt templates (diversification and simplification) = two prompt texts in Appendix D (Figures 7-8)
    The entire Adaptive Problem Restructuring mechanism is determined by these unvalidated prompts; no ablation, equivalence check, or ordering analysis of the rewrites is reported.
axioms (5)
  • ad hoc to paper Rewrite equivalence: f_p(q, d) preserves the mathematical content and answer a of the original problem q.
    Section 3.4 asserts 'the original answer a is strictly preserved' with no verification; the value filter in Eq. (8) checks only 0 < Acc(q', ...) < 1, which does not test equivalence. All training labels on restructured problems rest on this premise.
  • domain assumption Rollout accuracy with G=4 samples at temperature 1.0 is a reliable difficulty signal.
    Eq. (3) defines Acc as an empirical mean over 4 samples, giving a 5-level grid {0, .25, .5, .75, 1}; thresholds 0.3/0.7 then reduce the hard and medium curricula to single accuracy levels. The paper does not discuss this quantization.
  • domain assumption No contaminating overlap between DAPO-Math-17k training data and the in-domain evaluation sets.
    Section 4.1 uses MATH-500, Minerva-Math, Olympiad, AMC23, and AIME24 as in-domain tests; these are public competition and math sets, and DAPO-Math-17k is built from public math sources. No decontamination analysis is provided.
  • standard math Standard GRPO clipped surrogate objective and KL penalty assumptions (Eqs. (1) and (2)).
    Policy gradient with clipped importance ratios and reference-policy KL, taken from cited literature; unproblematic background.
  • domain assumption The verifier correctly judges correctness on original and rewritten problems.
    Eq. (3) treats Verifier(y_i, a) as ground truth; since rewrites keep the original answer a, any grader error or ambiguity in a rewritten problem differentially affects restructured samples.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning." pith.science (2026). https://pith.science/paper/HGXRU3A3

@misc{pith2026250925004,
  author       = {Pith},
  title        = {Pith review of: CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HGXRU3A3}},
  note         = {Machine review of arXiv:2509.25004}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems. We propose \textbf{CLPO (Curriculum Learning meets Policy Optimization)}, a self-evolving curriculum framework that uses on-policy rollout accuracy to identify solved, medium-difficulty, and hard problems, then restructures selected tasks according to the model's current capability. Hard problems are simplified to become learnable, while medium-difficulty problems are diversified to provide useful training variation. This allows the learning curriculum to co-evolve with the policy rather than remaining fixed as the model's capability boundary shifts. Rather than treating these rewrites as static data augmentation, CLPO optimizes restructuring trajectories with credit assigned by the downstream accuracy gain of the rewritten problem, requiring no additional human annotations beyond the original verifiable answers. Experiments across mathematical reasoning and out-of-domain general reasoning benchmarks show that CLPO substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points, respectively. Ablation studies on math and code domains further show that both the restructuring mode and the rewriting loss contribute to the final gains, demonstrating that CLPO provides a scalable and robust pathway for eliciting stronger reasoning capabilities through a self-evolving curriculum.

Figures

Figures reproduced from arXiv: 2509.25004 by Guanjun Jiang, Guohao Sun, Kevin Zhang, Rujun Guo, Shaoyu Liu, Shijie Zhang, Shiyu Liu, Wangxiao Zhao, Xiang Guo, Zheng Xiao.

Figure 1
Figure 1. Figure 1: An overview of the Guided Self-Evolution paradigm in our proposed framework CLPO [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall workflow of CLPO. The framework consists of three main stages detailed in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation studies on Adaptive Problem Restructuring. Left: Comparison of restructuring [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison of dynamic KL regularization with different scaling factors ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Test-time scaling performance (pass@k) on AIME2024 (left) and AIME2025 (right). The [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of the evolution of problem difficulty distribution during training for CLPO [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The prompt used for diversification restructuring of medium-difficulty problems. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The prompt used for simplification restructuring of hard-difficulty problems. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The original medium-difficulty problem presented in a symbolic style and the correspond [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The restructured version of the problem, elaborated into natural language by CLPO, and [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

    cs.CL 2026-06 unverdicted novelty 7.0

    ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...

  2. Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

    cs.LG 2026-07 conditional novelty 6.0

    OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

  3. ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents

    cs.AI 2026-06 unverdicted novelty 6.0

    ENVS generates verified supervision via environment-native search in OSWorld VMs to train GUI agents, reaching 30.3 pass@8 on 300 tasks while using less compute than ARPO baselines and introducing OSWorld-Noisy for in...

Reference graph

Works this paper leans on

27 extracted references · 22 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

  2. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

  3. [8]

    Train long, think short: Curriculum learning for efficient rea- soning.arXiv preprint arXiv:2508.08940,

    Hasan Abed Al Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long, think short: Curriculum learning for efficient rea- soning.arXiv preprint arXiv:2508.08940,

  4. [9]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

  5. [11]

    Beyond pass@ 1: Self-play with variational problem synthesis sustains rlvr.arXiv preprint arXiv:2508.14029,

    Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass@ 1: Self-play with variational problem synthesis sustains rlvr.arXiv preprint arXiv:2508.14029,

  6. [12]

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

    Preprint. V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. InInternational conference on machine learning, pp. 1928–1937. PmLR,

  7. [13]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

  8. [15]

    Efficient reinforcement finetun- ing via adaptive curriculum learning.arXiv preprint arXiv:2504.05520,

    Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning.arXiv preprint arXiv:2504.05520,

  9. [16]

    Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726,

    Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726,

  10. [19]

    Pass@ k policy optimization: Solving harder reinforcement learning problems.arXiv preprint arXiv:2505.15201,

    Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems.arXiv preprint arXiv:2505.15201,

  11. [20]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

  12. [21]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245,

    Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245,

  13. [22]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang

    Preprint. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

  14. [23]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

  15. [24]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

  16. [25]

    Critique-grpo: Advancing llm reasoning with natural language and numerical feedback.arXiv preprint arXiv:2506.03106,

    Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. Critique-grpo: Advancing llm reasoning with natural language and numerical feedback.arXiv preprint arXiv:2506.03106,

  17. [26]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

  18. [27]

    Table 3: CLPO-specific hyperparameters

    Software frameworks used. Table 3: CLPO-specific hyperparameters. Name Value Description hard threshold (τhard) 0.3 Accuracy threshold for hard problems. medium threshold (τmed) 0.7 Accuracy threshold for medium problems. hard kl scaler (λhard) 0.3 KL scaling factor for hard problems. non hard kl scaler (λnon-hard) 1.0 KL scaling factor for non-hard probl...

  19. [1998]

    Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

  20. [2009]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

  21. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

  22. [2018]

    Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient train- ing r1-like reasoning models.arXiv preprint arXiv:2503.17287,

    Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient train- ing r1-like reasoning models.arXiv preprint arXiv:2503.17287,

  23. [2020]

    Theoremqa: A theorem-driven question answering dataset.arXiv preprint arXiv:2305.12524,

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset.arXiv preprint arXiv:2305.12524,

  24. [2021]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

  25. [2023]

    Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

  26. [2024]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

  27. [2025]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.