Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By mixing a small number of previously verified correct reasoning paths into every GRPO update, RLEP makes math-reasoning RL converge faster and finish higher, gaining 1.7, 2.5, and 5.2 points on AIME-2024, AIME-2025, and AMC-2023.

desk verdict RLEP's replay recipe is plausible and cleanly described, but the headline 'faster and stronger' claims do not survive once you count the omitted 400-step seed run. read the letter →

arxiv 2507.07451 v1 pith:IRIPJL66 submitted 2025-07-10 cs.CL

Hongzhi Zhang , Jia Fu , Jingyuan Zhang , Kai Fu , Qi Wang , Fuzheng Zhang , Guorui Zhou This is my paper
classification cs.CL
keywords reinforcementlearningexperiencereplayLLMreasoningGRPOmathematicalpolicystabilitytrainingconvergenceverifiedrewards
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that replaying verified successful reasoning trajectories during reinforcement learning makes LLM math training both faster and stronger. After an ordinary RL run produces a seed policy, RLEP keeps every answer that passed the verifier and, at each update, mixes a handful of these old successes into the batch of freshly generated rollouts. The result, on a 7-billion-parameter math model, is that the policy reaches the tuned baseline's peak accuracy far sooner and then exceeds it: +1.7 points on AIME-2024, +2.5 points on AIME-2025, and +5.2 points on AMC-2023. If correct, RLEP is a near-free addition to existing GRPO-style training pipelines that attacks instability and weight drift without extra rollouts or new rewards.

What carries the argument

The load-bearing object is the experience pool plus the replay-blended GRPO update. At every update step the policy generates $G$ fresh trajectories, the method appends $M$ verified successes sampled from the pool (group size $G'=G+M$), and the token-mean, clip-higher GRPO objective is applied to the mixed group with advantages standardized over all $G'$ trajectories. This makes each update simultaneously reinforce proven chains and explore new ones, and that joint signal is what the paper claims produces faster convergence and a higher final performance ceiling.

What would settle it

Recompute the RLEP update with an off-policy correction that reweights replayed tokens by the ratio of the current policy's probability to the old policy's probability; if the corrected estimator removes the reported accuracy gains on AIME-2024, AIME-2025, or AMC-2023, then the mixed-batch GRPO update in Eq. (4) is the source of the improvement rather than the replayed content.

Watch

Extended reading notes

Core claim

RLEP's central claim is that the plateauing of reinforcement learning with verifiable rewards is not an information ceiling but a reuse problem: trajectories the policy already knows how to solve are abandoned, so each run re-explores from scratch. RLEP collects a pool of verifiably correct trajectories from a converged seed policy, then resumes training with mini-batches that blend fresh rollouts with randomly sampled high-quality replays. Advantages are computed over the enlarged mixed group, so replayed successes and new rollouts share one baseline. On three math benchmarks the replay run reaches the baseline's best accuracy in a fraction of the updates and ends higher, and the authors attribute both effects to replay shielding the policy from unproductive exploration and concentrating learning on promising paths.

Load-bearing premise

The load-bearing assumption is that old successful answers generated by the earlier seed policy can be fed into the GRPO estimator as if the current policy produced them, with no correction for the mismatch between the two policies.

Editorial extensions

If this is right

  • RLEP reaches the baseline's peak AIME-2024 accuracy by step 135 while the baseline takes 380 steps, and it surpasses the baseline's best AIME-2025 score after only 50 steps.
  • Final accuracy improves from 38.2% to 39.9% on AIME-2024, from 19.8% to 22.3% on AIME-2025, and from 77.0% to 82.2% on AMC-2023.
  • Because replay is applied uniformly across all prompts rather than only to difficult cases, the benefit extends to the whole training distribution instead of patching hard examples.
  • Per-step runtime increases by under five seconds relative to the baseline, so the reported gains come without materially longer training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the advantage is normalized over the enlarged mixed group, replayed successes mechanically raise the group mean and shrink the advantage assigned to fresh correct rollouts; an ablation that replays the same pool but computes advantages over fresh rollouts only would reveal how much of the gain is due to this re-baselining rather than to the replayed content.
  • Inference: the recipe needs only a rule-based verifier, so it should transfer to other verifiable-reward domains such as code generation or theorem proving; a direct test would run RLEP unchanged on a code benchmark and look for the same early-convergence signature.
  • Inference: the paper's null result for replaying failures leaves open a plausible extension: failure trajectories filtered by near-miss structure, such as a sound derivation with a wrong final step, might still teach the policy even if unfiltered errors are too heterogeneous to help.
  • Inference: the claimed reduction of weight drift is testable without new training by measuring parameter distance from the pretrained checkpoint across matched baseline and RLEP runs; the paper reports accuracy curves but no such distance diagnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RLEP, a two-phase reinforcement-learning-with-verifiable-rewards method for LLM reasoning. Phase 1 runs a standard GRPO-style baseline (DAPO-nodyn-bs64) for 400 PPO steps and uses the resulting seed policy to decode trajectories, retaining only rule-verified correct solutions in an experience pool. Phase 2 restarts from the base model and, at each update, mixes G fresh on-policy rollouts with M replayed success trajectories, computes token-level clipped GRPO advantages over the combined group (Eqs. (4)-(5)), and updates the policy. Experiments on Qwen2.5-Math-7B report faster convergence and higher final accuracy on AIME-2024, AIME-2025, and AMC-2023 relative to the tuned baseline. Code, datasets, and checkpoints are publicly released.

Significance. If the empirical claims held under proper controls, RLEP would be a simple, low-cost addition to existing GRPO pipelines: it requires no changes to the reward model or verifier, and it proposes a clean two-phase 'collect then restart with replay' design that differs from hard-prompt-rescue variants in the related work. The release of code, datasets, and checkpoints is a genuine strength for reproducibility. The significance is currently tempered by two load-bearing weaknesses: Eq. (4) applies an on-policy GRPO estimator to a batch containing off-policy replayed trajectories with no correction, and the reported comparisons are not compute-matched because the 400-step experience-collection run is omitted from the convergence and final-performance accounting. The contribution is plausible but not yet established; at present it is best viewed as a promising technical report rather than a validated method.

major comments (4)
  1. [§3.2, Eq. (4)] Equation (4) writes the objective as an expectation over trajectories drawn from π_θ_old, but the replayed trajectories in the mixed batch were generated by the seed policy used in the experience-collection phase, not by the current behavior policy. The ratio π_θ(o_i,t|q) / π_θ_old(o_i,t|q) is therefore not a valid importance-sampling correction for those trajectories, and the policy-gradient estimate is biased. Because replayed trajectories are all correct and typically receive high rewards, the bias is likely to inflate the replay signal rather than vanish. The paper should either incorporate a proper off-policy correction (for example, importance weights with clipping, or a separate conservative objective for replayed data) or provide an empirical diagnostic showing that the bias is negligible, such as comparing Eq. (4) with a corrected estimator on the same batch, or ablating how replayed advantages are normalized. Without this, the mechanism claimed for RLEP is not supported by the paper's own equations.
  2. [§4.2] The compute accounting is incomplete. The experience pool is constructed by running DAPO-nodyn-bs64 for 400 PPO steps, and the replay run then restarts from the base model. Thus reaching the baseline peak on AIME-2024 at replay step 135 costs at least 400 + 135 = 535 total PPO steps, and the final reported accuracies correspond to about 400 replay steps on top of the 400-step seed run. The baseline is trained for only 400 steps, and no control is reported, such as a baseline continued for another 400 steps or a baseline trained for the same total number of updates. With the current design, the higher final accuracy of RLEP is confounded with additional total compute, and the 'substantially fewer updates' claim in the abstract refers only to replay-phase steps. A compute-matched comparison is needed before 'faster convergence' or 'stronger final performance' can be attributed to experience replay rather than to additional training.
  3. [§4.2, Figure 4] The main results are based on a single run with one base model (Qwen2.5-Math-7B) and no error bars or seed-to-seed variance. The text explicitly reports the best points on AIME-2024 and AIME-2025 as 'peak value of the line' (second bullet). Given that the reported differences are small on AIME-2024 (+1.7pp) and AIME-2025 (+2.5pp), and that the benchmarks contain a limited number of problems, it is not possible to assess whether these differences are stable or within run-to-run noise. At minimum, the authors should report the mean and standard deviation over at least three seeds, and ideally full training curves with confidence bands, for the central RLEP-versus-baseline comparison.
  4. [§3.2, Eq. (5)] Equation (5) standardizes advantages over the mixed group of G' trajectories that contains both fresh rollouts and replayed successes. Because the replayed trajectories are all correct, their presence raises the group mean and changes the baseline used for the fresh rollouts. This is a design choice rather than an outright error, but its interaction with the off-policy issue in Eq. (4) is not analyzed. If replay is intended to provide a common baseline, the paper should at least compare with a variant in which replayed trajectories are excluded from the mean and standard deviation but still included in the policy-update loss, or otherwise show that the mixed-batch standardization does not distort the advantages assigned to fresh rollouts.
minor comments (5)
  1. [§4.1] There are several typographical errors: 'wile' should be 'while', 'dynmic sampling' should be 'dynamic sampling', and 'we therefore use the configure' should be 'we therefore use the configuration'.
  2. [§4.2] The text first says 'For every question we maintain an experience pool' and later says 'randomly sample M successful trajectories from the experience pool'; please clarify whether each prompt has a separate replay pool and how the pool size varies across prompts when the minimum-correct-paths threshold is applied.
  3. [§4.2] The wall-clock cost of the experience-collection phase is not reported, even though the paper says 'per-step runtime increases by under 5s' for the replay phase; reporting total wall-clock time for both phases would help readers evaluate the compute-matched concern.
  4. [§2] Since LUFFY is cited as addressing off-policy guidance with a redesigned advantage estimator, a brief comparison with that approach, or an explanation of why the same correction is not needed here, would help position RLEP's claim of novelty.
  5. [References] Reference [11] is listed as 'HybridFlow: A flexible and efficient rlhf framework,' but the text refers to 'Verl (Sheng et al., 2024)'; please make the framework name and citation title consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RLEP is an empirical method evaluated on external benchmarks.

full rationale

RLEP's pipeline (Sec. 3.2) collects verified trajectories from a seed RL run and then applies the standard token-level GRPO objective (Eq. 4) to mini-batches that mix fresh rollouts with replayed successes. There is no fitted parameter that is later relabeled as a prediction: the reported AIME-2024/2025 and AMC-2023 accuracies are measured externally and are not quantities that Eq. (4) defines or forces. The experience pool is built from baseline rollouts, but that is data sourcing rather than a mathematical reduction; nothing in the construction of the pool or the objective makes the held-out accuracy gains true by definition. The DAPO-related citations with overlapping authors are used as an algorithmic starting point and as a comparison baseline, not as a load-bearing justification of RLEP's central claim, and no uniqueness theorem or ansatz is imported from prior work. Concerns about compute matching (the 400-step seed run omitted from the 'fewer updates' comparison) and about the off-policy application of GRPO to replayed trajectories are validity/experimental-design issues, not instances of circularity. No specific Eq. X = Eq. Y reduction or fitted-input-as-prediction step can be exhibited, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces no new objects beyond a stored set of trajectories. The free parameters listed are the hand-set knobs of the two-phase recipe; the most consequential is the implicit assumption that stale-policy trajectories can be replayed in an on-policy GRPO update without correction.

free parameters (6)
  • replay_count_M = 2
    Number of replayed successful trajectories added per question in RLEP phase; no ablation or search reported. Controls the balance between exploration and replay.
  • fresh_rollout_count_G = 16
    Fresh on-policy rollouts per question in RLEP phase; differs from baseline's larger rollout budget, so part of the effect could come from batch composition.
  • collection_sample_count = 64
    Number of candidate answers sampled per question during experience collection; defines the size of the experience pool and the hidden compute cost.
  • minimum_correct_paths = >=2
    Filtering rule for keeping questions in the pool; may change the distribution of replayed training data.
  • collection_temperature = 0.7
    Decoding temperature used to build the experience pool, with top-p 0.95; differs from the training temperature 1.0.
  • seed_policy_steps = 400
    Length of the preliminary DAPO-nodyn-bs64 run used to collect experiences; not counted in the reported runtime overhead.
assumptions (4)
  • standard math Standard GRPO with token-mean and clip-higher is a valid policy-gradient estimator when all samples come from the current behavior policy.
    Section 3.1 reviews GRPO and its DAPO refinements; Eq. (4) inherits this machinery.
  • ad hoc to paper Replayed trajectories from a stale seed policy can be plugged into Eq. (4) as if they were on-policy samples from pi_theta_old.
    Section 3.2 and Eq. (4) make this assumption implicitly; no off-policy correction is offered, which is the main technical weakness.
  • domain assumption Rule-based verification correctly identifies correct reasoning paths for all training and evaluation questions.
    Section 4.2 relies on 'verified as correct by the reward model' to build the experience pool and to score AIME/AMC results.
  • domain assumption Evaluation benchmarks are disjoint from the training distribution.
    AIME-2024/2025 and AMC-2023 are treated as held-out sets; the paper does not discuss contamination.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning." pith.science (2026). https://pith.science/paper/IRIPJL66

@misc{pith2026250707451,
  author       = {Pith},
  title        = {Pith review of: RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRIPJL66}},
  note         = {Machine review of arXiv:2507.07451}
}
read the original abstract

Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present \emph{RLEP}\, -- \,Reinforcement Learning with Experience rePlay\, -- \,a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.

Figures

Figures reproduced from arXiv: 2507.07451 by the authors.

Figure 2
Figure 2. b gives an overview of the RLEP training workflow. During each [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Experience Augmented Policy Optimization for LLM Reasoning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    EAPO reuses prior RL policy experience adaptively at decision points in LLM rollouts with adapted importance sampling and reports gains over prior RLVR methods on math benchmarks.

  2. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

  3. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages · cited by 3 Pith papers

  1. [1]

    C. An, Z. Xie, X. Li, L. Li, J. Zhang, S. Gong, M. Zhong, J. Xu, X. Qiu, M. Wang, and L. Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models. URL https://hkunlp.github.io/blog/2025/Polaris

    work page 2025

  2. [2]

    B. R. Bartoldson, S. Venkatraman, J. Diffenderfer, M. Jain, T. Ben-Nun, S. Lee, M. Kim, J. Obando-Ceron, Y. Bengio, and B. Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable llm post-training, 2025. URL https://arxiv.org/abs/2503.18929

    arXiv 2025

  3. [3]

    S. Dou, M. Wu, J. Xu, R. Zheng, T. Gui, Q. Zhang, and X. Huang. Improving rl exploration for llm reasoning through retrospective replay, 2025. URL https://arxiv.org/abs/25 04.14363

    work page 2025

  4. [4]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P . Wang, X. Bi, et al. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  5. [5]

    L.-J. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 8(3–4):293–321, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992699. URL https://doi.org/10.1007/BF00992699

    doi:10.1007/bf00992699 1992

  6. [6]

    Z. Liu, C. Chen, W. Li, P . Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    arXiv 2025

  7. [7]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529–533, 2015. URL https://api.semanticsc...

    work page 2015

  8. [8]

    Openai o1 system card

    OpenAI. Openai o1 system card. https://openai.com/index/openai-o1-system-c ard/, Dec. 2024. Updated December 5, 2024

    work page 2024

Show all 17 references
  1. [9]

    Schaul, J

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay.arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  2. [10]

    Z. Shao, P . Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  3. [11]

    Sheng, C

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 , 2024. 8

    2024 arXiv

  4. [12]

    C. Wang, L. Wei, Y. Zhang, C. Shao, Z. Dan, W. Huang, Y. Wang, and Y. Zhang. Eframe: Deeper reasoning via exploration-filter-replay reinforcement learning framework, 2025. URL https://arxiv.org/abs/2506.22200

    2025

  5. [13]

    S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025

    2025 arXiv

  6. [14]

    J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang. Learning to reason under off-policy guidance, 2025. URL https://arxiv.org/abs/2504.14945

    2025 arXiv

  7. [15]

    A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  8. [16]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. L...

    2025 arXiv

  9. [17]

    Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 9

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.