{"id":"0c7a3fb9-2d95-4186-a66c-f726b4376c9e","arxiv_id":"2504.14363","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RRL replays value-model-selected promising intermediate states during LLM RL training, preserving exploration and improving performance on code, math, and RLHF tasks.","lead":"RRL is a new reinforcement learning algorithm that saves promising intermediate solution states from early training and later replays them so the LLM can revisit and finish those ideas. It reports gains over standard PPO on code, math, and dialogue tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are not isolated from canonical-solution replay: CS-only replay already beats PPO by +2.8 points, and no random-prefix replay control shows that value-selected states drive exploration.","rationale":"The reader's weakest assumption was that the value model's argmax state is genuinely promising. That is part of the problem, but the more load-bearing gap is experimental: the design never isolates the exploration mechanism from the act of replaying any prefix, and especially from replaying canonical solution prefixes. The 'States only from CS' ablation produces 34.5% versus 31.7% for PPO, so a substantial share of the headline gain can be explained by exposing the model to correct-solution prefixes, which is not the exploration story the paper tells. The 'States only from PGS' arm is also uninformative without a random-prefix replay control, because it does not show that value-model selection improves over arbitrary continuation training from on-policy prefixes. The Limitations section's own admission of value-model instability reinforces that the mechanism is unvalidated, and the inconsistent optimization equation in Section 2.2 makes the exact procedure impossible to reconstruct. These concerns do not prove the method is ineffective, but they do mean the central claim is not yet established. Since the reader already assigned a CONDITIONAL verdict for similar reproducibility and evidence reasons, my assessment does not move the verdict; it sharpens the specific missing control.","tokens_in":11059,"tokens_out":6808,"duration_ms":68828,"concrete_test":"Run the APPS+ experiment with the replay schedule and compute held fixed, but replace value-argmax state selection with random selection among intermediate states of policy-generated solutions; add a second arm that replays random prefixes of canonical solutions. If random-prefix replay matches the reported PGS-only 34.2% or CS-only 34.5%, then value-model selection and the exploration mechanism are not the cause. If value-selected replay clearly outperforms random-prefix replay at matched compute, the concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that replaying value-model-selected promising intermediate states improves exploration enough to outperform vanilla PPO. The paper's own ablation data undercut this attribution. In Table 1, 'States only from CS' reaches 34.5% on APPS+ versus 31.7% for vanilla PPO, meaning that replaying canonical-solution prefixes alone recovers most of the +3.5% full-RRL gain; 'States only from PGS' reaches 34.2%. Neither arm is a clean test of exploration: both add a replay schedule that PPO lacks, and the CS arm also injects ground-truth solution prefixes, which is closer to imitation or curriculum learning than to the proposed 'return to a promising state and explore' mechanism. The paper's Limitations section explicitly admits that value-model-selected states can contain errors and that the value model is unstable and inaccurate early in training, so the premise that the argmax state is genuinely promising is asserted rather than demonstrated. A control that replays random intermediate states at the same frequency and compute budget is missing. Without such a control, the reported gains cannot be attributed specifically to the exploration mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a failure mode in RL for LLM reasoning: policy gradient optimization suppresses early-trajectory prefixes that are promising but incomplete, thereby reducing exploration in later training stages. The proposed method, Retrospective Replay-based Reinforcement Learning (RRL), stores the highest-value states from policy-generated and canonical solutions in a buffer and replays them with probability p during training, increasing replay probability as training progresses. Experiments are reported on APPS+ code generation, GSM8K and MATH reasoning, and HH-RLHF dialogue; headline results are APPS+ overall 35.2% vs 31.7% for vanilla PPO, GSM8K 70.7% vs 68.8%, MATH 34.3% vs 33.3%, and 55.4% win rate over PPO in dialogue. Ablations show that replaying states only from canonical solutions reaches 34.5% and states only from policy-generated solutions reaches 34.2% on APPS+.","tokens_in":11317,"tokens_out":4832,"duration_ms":43201,"significance":"The underlying observation is timely and plausible: RL training can cause LLMs to abandon partially correct solution prefixes that later, more capable checkpoints might be able to complete. RRL is a simple, modular addition to PPO and the experiments cover three distinct task families, which is a strength. The paper also provides explicit ablations, a sensitivity analysis, and an honest limitations section, and it states the exact evaluation protocols. However, the central attribution—that value-model-selected states improve exploration specifically—is not yet established, because the paper lacks a random-prefix replay control and the value-model premise is admitted to be unreliable early in training. In addition, the stated optimization objective in Eq. (3) is not the PPO objective referenced in the text, and results are reported without seeds, error bars, or code. If these load-bearing issues are resolved, the method could be a useful contribution to RL post-training for LLMs.","major_comments":[{"comment":"Eq. (3) is not a PPO objective. It lacks the importance ratio and clipping used in PPO and does not include the GAE advantage A_i from Eq. (1), yet the text states that 'we utilize the proximal policy optimization (PPO) algorithm.' The paper should give the exact surrogate objective used for both the normal-exploration branch and the replay branch, including how GAE advantages are computed for rollouts that start from replayed intermediate states and how the token masking of replayed states is implemented in the loss. As written, Eq. (3) cannot reproduce the claimed training procedure.","section":"§2.2, Eq. (3)"},{"comment":"The ablations undermine the central attribution of the gains to value-selected exploration. 'States only from CS' (34.5%) and 'States only from PGS' (34.2%) already outperform vanilla PPO (31.7%) by roughly 2.8 and 2.5 points, respectively, recovering most of the full RRL gain of 3.5 points. Both ablation arms add a replay schedule that PPO lacks, and the CS arm also injects ground-truth solution prefixes, which is closer to curriculum learning or imitation than to the proposed 'return to a promising state and explore' mechanism. The paper's own Limitations section admits that value-model-selected states may contain errors and that the value model is unstable and inaccurate early in training, so the premise that the argmax state is genuinely promising is asserted rather than demonstrated. A control that replays random intermediate states at the same replay frequency, buffer capacity, and compute budget is missing. Without such a control, the reported gains cannot be attributed specifically to the value-model-based exploration mechanism; this should be added and analyzed, for example by reporting the success rate of completions from replayed states versus random prefixes.","section":"Table 1; §5 Limitations"},{"comment":"The statistical evidence is not yet sufficient for the claimed improvements. No seeds, confidence intervals, or code are reported. Several headline deltas are small, notably MATH 34.3% vs 33.3% and GSM8K 70.7% vs 68.8%, and the dialogue comparison against PPO is based on 100 pairwise evaluations (31 wins, 44 ties, 25 losses). The paper should report results over at least three seeds with standard deviations, or provide significance tests or confidence intervals, especially for the smaller math and dialogue gains.","section":"Tables 1 and 2; §3.2"}],"minor_comments":[{"comment":"The replay probability is defined as 'p = beta * (Step/Step num of one epoch) if Epoch = 1 else beta', but no epoch structure is otherwise defined in the algorithm or training loop; please clarify the schedule and how 'Step num of one epoch' is computed.","section":"Algorithm 1, line 2"},{"comment":"The sensitivity analysis does not state which dataset or task Figure 3 refers to, and the y-axis label 'Accuracy' is not tied to the Pass@1 metric used in Table 1; the vanilla PPO baseline of 27.9 also differs from the APPS+ overall value of 31.7 in Table 1, so the setting should be specified.","section":"Figure 3"},{"comment":"There are small presentation errors: 'Deeseek-Coder-Instruct' should be 'DeepSeek-Coder-Instruct', and the text says 'an improvement of 1.9 PPL' where 'PPL' should be 'percentage points' or similar.","section":"§3.1"},{"comment":"The in-text references 'Table 3.1' and 'Table 3.2' do not match the actual table numbering (Table 1 and Table 2); please align the cross-references.","section":"Table references in §3.2"},{"comment":"Several RRL-specific hyperparameters are not reported: the maximum buffer capacity of five is mentioned in §2.2, but the counter update rule, the value-model stabilization threshold, and the epoch-dependent replay schedule are not specified with concrete values; please include these for reproducibility.","section":"§3.1 / Implementation"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the proposed mechanism is interesting, but the current evidence is not sufficient to support the strong exploration-attribution claim. The missing random-prefix replay control and the inconsistency of the stated objective with the actual PPO procedure are both fixable with additional experiments and rewriting; therefore major_revision rather than reject. If the authors can add the control, correct the objective, and provide multi-seed results with code release, the paper could become a solid contribution to the RL-for-LLMs literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you work on RL post-training for LLMs: it's a clean Go-Explore-inspired twist—use the value model to pick the highest-value intermediate state from policy-generated and canonical solutions, store in a bounded buffer, and replay them with a masking loss as training proceeds. That combination is new in LLM RL, and the paper tests it across code, math, and RLHF. The best results beat vanilla PPO on APPS+ (35.2 vs 31.7), GSM8K/MATH, and RLHF win rate. The intuition that policy gradient suppresses promising but incomplete early trajectories is plausible and matches diversity-collapse observations.\n\nWhat the paper does well: the method is simple, related work is honestly positioned (Go-Explore, experience replay, curriculum RL), and there is a limitations section that openly admits the value model can be unstable and select error-containing states. It also runs a sensitivity analysis on the replay coefficient and includes ablations separating canonical-solution states from policy-generated states.\n\nThe soft spots are real. First, the objective in Eq. 3 is not the PPO objective described in the text; the paper never derives how GAE advantages connect to that loss. That matters for reproducibility. Second, no seeds, no error bars, and no code are provided; the headline gains are single-run numbers. Third, and most important for the central claim: the ablation table cuts against the exploration story. Replaying only canonical-solution states already gives 34.5% on APPS+, most of the full-RRL gain, and that arm injects ground-truth prefixes—closer to imitation/curriculum than to 'return to a promising state and explore.' Neither arm controls for simply giving PPO extra replay schedules or extra compute. Without a random-prefix replay control, the specific attribution to value-model-selected states is not established. The limitations passage about early value-model unreliability makes this more pressing, not less. That said, the paper's overall direction is reasonable and the full method does beat the ablations, so I don't read this as a fundamental flaw in the idea—just in the current evidence.\n\nWho is this for? Researchers working on exploration in LLM RL will find the idea worth engaging. It deserves a serious referee, but it needs fixing: rewrite Eq. 3 with a proper objective, release code and seeds, and add the missing control.","headline":"RRL is a sensible Go-Explore-style replay idea for LLM RL, but the paper currently under-delivers on evidence: the optimization objective doesn't parse, and the ablations don't isolate the exploration mechanism.","tokens_in":11797,"tokens_out":2582,"would_cite":true,"duration_ms":23638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that policy gradients in LLM RL suppress promising early states, and that replaying those states restores exploration and improves reasoning.","keywords":["reinforcement learning","LLM reasoning","exploration","replay buffer","policy gradient","value model","code generation","RLHF"],"falsifier":"Run RRL with buffer states selected uniformly at random from the same generated and canonical solutions instead of by $\\arg\\max_{s_i} V(s_i)$; if the random-state version matches RRL's APPS+ or MATH gains, value-model selection is not the component that improves exploration.","tokens_in":10878,"feed_emoji":"🔁","tokens_out":6200,"duration_ms":52525,"temperature":0.7,"pith_summary":"The paper identifies a failure mode in RL fine-tuning of large language models: for hard problems, the early model finds promising partial ideas but gets negative reward because the full solution is wrong, so the policy gradient suppresses those partial ideas. Later in training the model is stronger but no longer explores them, and the paper argues this lost exploration, not just sampling diversity, is why RL stalls on complex reasoning. To fix it, RRL stores a high-value intermediate state from each failed generated solution and from the canonical solution, then replays those states as starting points with a tunable probability. In the paper's experiments this raises pass@1 on APPS+ from 31.7% to 35.2%, GSM8K from 68.8% to 70.7%, MATH from 33.3% to 34.3%, and gives a 55.4% win rate over vanilla PPO in dialogue RLHF.","feed_headline":"Replaying early promising states boosts LLM RL on code and math","feed_subtitle":"RRL resumes from high-value intermediate states and beats vanilla PPO on APPS+, GSM8K, MATH, and RLHF.","key_machinery":"The promising-state buffer is the mechanism. For a problem $q$, RRL computes $s^* = \\arg\\max_{s_i} V(s_i)$ on the generated solution and, if the solution is wrong, on the canonical solution too; each stored state has a counter, buffer capacity is five per problem, and solved states are removed. The replay coefficient $p$ controls how often generation starts from a buffered state instead of the problem prompt, with $p$ growing over the epoch, and the loss masks state tokens so the model is not over-optimized on already-written text. This is what lets a later, stronger policy return to an earlier promising prefix and continue exploring from it.","core_discovery":"The central claim is that the standard policy-gradient objective actively erases good prefixes: since a solution is scored only by final correctness, a trajectory with a correct idea and one bad token gets negative advantage, and every token in it, including the good prefix, is down-weighted. Over training this is compounded by reduced action diversity, so the strengthened model cannot rediscover states the weak model once visited. RRL counters by using the value model to identify the highest-value state in both the policy-generated (possibly wrong) solution and the canonical solution, storing these states in a bounded buffer and re-generating completions from them while masking the stored state tokens in the loss. The paper reports consistent gains over vanilla PPO and prior RL methods, with the largest relative improvements on the hardest APPS+ splits (Interview 20.1% to 23.5%, Competition 5.0% to 7.9%).","pith_inferences":["A natural test of the mechanism is to ablate the value model itself: replacing the argmax state with a random prefix from the same failed solution would show whether value-based selection or mere resumption drives the gains.","The paper's limitation that early value estimates are unreliable suggests that adding execution feedback for code or verifier scores for math to filter replayed states before storing them could further reduce the risk of replaying errors; the paper leaves this for future work.","RRL is algorithmically close to experience replay but with a different target, replaying states to resume exploration rather than to update value estimates; this distinction could be tested by comparing RRL against prioritized experience replay on the same LLM tasks."],"forward_implications":["If the paper's diagnosis is right, any sparse-reward RL post-training of LLMs should show the same early-idea suppression, so RRL-style replay should transfer to new reasoning benchmarks beyond APPS+, GSM8K, and MATH.","The ablations indicate that replaying only policy-generated states or only canonical states helps less than the combination, so the full method's advantage is specifically the mix of imperfect model ideas and correct reference prefixes.","The RLHF result suggests that preserving exploration during alignment improves both helpfulness and safety, not only objective reasoning accuracy.","The sensitivity analysis shows gains at replay coefficients 0.1 to 0.5, with 0.1 optimal, so the method is not tied to a single hyperparameter setting."],"supporting_citations":[{"why":"Supplies the PPO optimizer that RRL wraps; vanilla PPO is the main baseline.","marker":"[37]"},{"why":"Defines the policy-gradient update whose suppression of promising but incorrect trajectories is the paper's motivating mechanism.","marker":"[42]"},{"why":"Provides the GAE advantage estimator used in the policy loss.","marker":"[36]"},{"why":"Introduces the APPS dataset and the code-generation evaluation.","marker":"[20]"},{"why":"A prior RL-from-compiler-feedback method on APPS+ that RRL compares against and extends.","marker":"[11]"},{"why":"GSM8K math benchmark, one of the two math evaluation sets.","marker":"[7]"},{"why":"MATH benchmark, the harder math evaluation set.","marker":"[21]"},{"why":"HH-RLHF dialogue dataset and RLHF setup on which RRL reports win rates.","marker":"[2]"}],"fun_headline_variants":["RRL replays early state gems to improve LLM reasoning","Policy gradient erases good prefixes; replay recovers them","Retrospective replay upgrades RL exploration for LLM math and code","Reusing early high-value states improves LLM RL and RLHF","RRL: Remember your good early ideas for better LLM RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The value model's highest-scoring state inside a mostly wrong solution is a genuinely good place to resume, so continuing from that state teaches more than starting from the problem prompt.","fun_headline_variants_meta":{"raw":{"variants":["RRL replays early state gems to improve LLM reasoning","Policy gradient erases good prefixes; replay recovers them","Retrospective replay upgrades RL exploration for LLM math and code","Reusing early high-value states improves LLM RL and RLHF","RRL: Remember your good early ideas for better LLM RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1650,"prompt_tokens":977,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":585}},"tokens_in":593,"tokens_out":673,"duration_ms":6355,"temperature":1.0,"reasoning_tokens":585,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:49:42.206474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RRL with buffer states selected uniformly at random from the same generated and canonical solutions instead of by $\\arg\\max_{s_i} V(s_i)$; if the random-state version matches RRL's APPS+ or MATH gains, value-model selection is not the component that improves exploration.","supporting_citations":[{"cited_title":"Advances in neural informa- tion processing systems12 (1999)","cited_arxiv_id":null,"evidence_quote":"Defines the policy-gradient update whose suppression of promising but incorrect trajectories is the paper's motivating mechanism."},{"cited_title":"In: Vanschoren, J., Yeung, S","cited_arxiv_id":null,"evidence_quote":"Introduces the APPS dataset and the code-generation evaluation."},{"cited_title":"In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"A prior RL-from-compiler-feedback method on APPS+ that RRL compares against and extends."}],"review_version":1}