{"id":"6a6b835d-c238-4912-874f-af062f35ebd7","arxiv_id":"2508.13023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"G2RPO-A adaptively shortens or lengthens the ground-truth reasoning guidance injected into GRPO rollouts, yielding mostly higher math and code accuracy for small language models than vanilla GRPO.","lead":"Small AI language models usually improve little from reinforcement learning. A new scheme gives some practice attempts a guiding hint and automatically adjusts hint length as the model improves, yielding better math and coding scores in most tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Guidance ratio α is selected on evaluation benchmarks, so the reported G2RPO-A gains may reflect tuning rather than the adaptive mechanism.","rationale":"The reader's weakest assumption focuses on Eq. (5) and whether reward fluctuations reflect true learning progress. That is a legitimate concern about mechanism robustness. However, the reader's own reported evidence includes the paper's Section 4.3 admission that reward variations can be confounded with batch difficulty, and the paper adds a curriculum to patch it. My review agrees that this is a real soft spot, but I find the more load-bearing and more immediately checkable threat is that α is tuned on the evaluation benchmarks themselves: the paper's own ablation tables 8 and 9 identify the per-model α by accuracy on the same MATH500/GPQA/HumanEval/LCB benchmarks that appear in the final tables. That creates a selection-on-evaluation confound that undermines the headline 'markedly surpasses' claim. The adaptive rule is still the intended contribution, and the paper does provide useful ablations (Tables 10, 11) comparing fixed guidance lengths and RDP versus G2RPO-A. Yet those ablations also reuse the tuned α, so they do not independently validate the adaptive controller under a fair hyperparameter regime. I do not see the concern as fatal: the method is clearly described, the code is released, and the experiments are substantial. With a pre-registered or held-out α, the claims could be accepted. For that reason the verdict should remain CONDITIONAL, not REJECT or UNVERDICTED.","tokens_in":14216,"tokens_out":1647,"duration_ms":14331,"concrete_test":"Re-run the Qwen3-0.6B and Qwen3-1.7B math and code experiments with a single, prespecified guidance ratio (e.g., α=0.5 for all models and tasks), initial guidance length 3072, and the same curriculum ordering used in the paper. If G2RPO-A still outperforms vanilla GRPO on the majority of benchmarks in Tables 3-5, the adaptive mechanism survives without test-set tuning. If the advantage shrinks or reverses on several benchmarks, the headline gains are attributable to α selection rather than to the adaptive controller.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that G2RPO-A markedly and consistently surpasses vanilla GRPO on math and code benchmarks. What would have to be true is that the observed gains come from the adaptive guidance mechanism (Eq. 5) plus curriculum ordering, and not from selection of the per-model guidance ratio α on the test benchmarks themselves. Inspection of Tables 8 and 9 shows α is chosen per model and per task family as the best-performing value on the evaluation benchmarks. For example, Qwen3-0.6B on math uses α=0.75 because it scores highest on MATH500/accuracy among five tested values, and Qwen3-1.7B on code uses α=1 because it scores highest on HumanEval and LiveCodeBench. The same benchmark values then appear in the headline Tables 3-5, with no held-out validation of α. This is selective-tuning on the test set: the reported gains over GRPO are upper bounds conditional on α having been tuned to the same benchmark, and the 'adaptive guidance helps' conclusion is confounded with 'the right fixed intervention ratio helps.' Because α and the initial guidance length 3072 are fixed hyperparameters, the design has 4+ tuned constants, and the paper does not show that an untuned, α-default version of G2RPO-A still beats GRPO. This is not internal inconsistency: the mechanism may be genuine. But the claim that 'G2RPO-A adaptively improves performance' requires the comparison to hold without per-benchmark α selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Guided GRPO, which injects ground-truth reasoning traces into a subset of rollouts for small language models, and identifies two design axes: the fraction of guided candidates (guidance ratio α) and the guidance length (ℓ). It reports that naive fixed-length guidance is not helpful, and that the optimal guidance length is context-dependent. The proposed method, G2RPO-A, adjusts the guidance length at each training step via Eq. (5), which increases ℓ when recent rewards fall and decreases it when rewards rise, while a curriculum-learning ordering of the training data is used to stabilize the reward signal. Experiments on math (MATH500, Minerva, GPQA, AIME24/25) and code (HumanEval, LiveCodeBench) benchmarks with Qwen3-0.6B/1.7B/8B, DeepSeek-Math-7B, and DeepSeek-Coder-6.7B claim that G2RPO-A markedly and consistently outperforms vanilla GRPO. The paper also includes ablations over α, fixed guidance lengths, decay policies, and curriculum ordering.","tokens_in":14534,"tokens_out":5024,"duration_ms":52052,"significance":"If the reported gains are genuine, the paper makes a useful practical contribution: an adaptive guidance-length controller that mitigates the small-model RLVR reward-sparsity problem, together with an open-source implementation. The systematic ablation of guidance ratio and guidance length is a strength, as is the attempt to isolate the effect of curriculum ordering. The central claim, however, is currently not established to the advertised strength. The main evidence is an empirical comparison in which the guidance ratio is selected on the evaluation benchmarks themselves, and in which several benchmark–model pairs show G2RPO-A underperforming vanilla GRPO. Multiple seeds and a held-out or default-α comparison are needed before the claim of consistent and substantial gains can be accepted. The adaptive controller's premise that reward trends reflect competence is also not independently validated. The novelty relative to concurrent adaptive-guidance work (e.g., Nath et al., 2025) is not demonstrated experimentally.","major_comments":[{"comment":"The guidance ratio α is tuned on the evaluation benchmarks themselves. Tables 8 and 9 report accuracy on MATH500, Minerva, GPQA, HumanEval, and LiveCodeBench for several α values, and the paper then uses the best-performing α per model and task family in the headline results of Tables 3–5. This is test-set tuning: the reported gains of G2RPO-A over vanilla GRPO are confounded with the effect of choosing a favorable fixed intervention ratio. To support the claim that adaptive guidance improves performance, the authors should either (a) report results with a fixed default α (e.g., α=1/4 or α=1/2) without per-benchmark selection, or (b) split the benchmarks into a tuning set and a held-out evaluation set and show that the selected α transfers.","section":"Section 5.1, Tables 3–5 and Tables 8–9"},{"comment":"The claim that G2RPO-A 'markedly surpasses vanilla GRPO on nearly every benchmark' and 'consistently boosts accuracy' is contradicted by several reported pairs. For example, Table 3 shows Qwen3-0.6B on MATH500 at 51.77 vs. GRPO 54.26; Qwen3-1.7B on Minerva at 15.10 vs. 16.19; Table 4 shows Qwen3-1.7B on MATH500 at 91.69 vs. 94.52; and Table 5 shows Qwen3-8B on HumanEval at 80.33 vs. 81.48 and Qwen3-1.7B on LiveCodeBench at 51.96 vs. 53.14. The paper should report all benchmark–model pairs with the sign of the difference and quantify the fraction of pairs where G2RPO-A wins, rather than relying on 'nearly every' and 'consistently.'","section":"Section 5.2, Tables 3, 4, 5, and Conclusion"},{"comment":"The adaptive controller assumes that a rise or fall in the current batch reward r_k is a faithful signal of the model's learning state. The paper itself flags in Section 4.3 that when consecutive batches differ in difficulty, reward variation no longer reflects true learning progress, and it mitigates this with a hand-chosen curriculum tier order. This premise is load-bearing but not independently verified. I suggest an additional control experiment: replace the reward-driven controller in Eq. (5) with (i) a random schedule constrained to the same guidance-length range and (ii) a schedule driven by rewards from a small held-out set; if the advantage of G2RPO-A over vanilla GRPO disappears against control (ii), the mechanism is not identifiable from the effect of a generally helpful intervention schedule.","section":"Section 4.3, Eq. (5)"},{"comment":"No seed information, variance estimates, or confidence intervals are reported for any experiment. With training sets of 1,000 problems for 5 epochs and the small absolute differences seen in several benchmark pairs (e.g., Table 3 GPQA for Qwen3-0.6B: 30.39 vs. GRPO 24.51; Table 5 LiveCodeBench for Qwen3-1.7B: 51.96 vs. 53.14), a single run is insufficient to support the 'markedly surpasses' wording. At minimum, the authors should report the number of seeds and either standard deviations or per-run numbers.","section":"Section 5.1 and all result tables"}],"minor_comments":[{"comment":"The displayed objective for Guided GRPO has unmatched delimiters: the sum over i contains a \\(\\frac{1}{|o_i|+|g_i|}\\) factor whose braces are not closed, making the equation difficult to parse. Please rewrite it with clear brackets for the guidance-token and output-token sums.","section":"Section 4.1, guided objective"},{"comment":"The parameters of the decay policies are not specified: the values of β, γ, and s used in the Table 2 experiments are not given, so the reader cannot reproduce the concave, linear, and stepwise decay curves.","section":"Section 4.2, Eq. (4)"},{"comment":"The abbreviation RDP is used in Tables 10 and 11 but is not defined in the main text; also, the text says that T=2 is 'already sufficient' and cites Tables 10 and 11, but those tables show guidance-length ablations rather than a sweep over history-window T. Either add a T-sweep table or correct the reference.","section":"Tables 10 and 11"},{"comment":"The table contains typos ('Minarva' for Minerva) and labels 'Random' and 'CL' without explaining how the random order was generated or whether it is matched to the CL order in content. Please clarify.","section":"Table 6"},{"comment":"The manuscript references an appendix with 'Detailed equations' and 'Appendix Figure 6,' but no appendix is present in the provided text. The full version must include the appendix or remove these references.","section":"General (appendix references)"}],"recommendation":"major_revision","confidential_remarks":"The central issue is benchmark-driven selection of α: the ablations in Tables 8–9 are on the same benchmarks as the headline results, so the core claim is not yet supported. If the authors add a held-out or default-α comparison and report multi-seed results, the contribution would likely be acceptable as a practical RLVR method for small models. Please also verify that the appendix is present in the published version, since several statements rely on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on G2RPO-A. The adaptive guidance-length controller in Eq. (5) is a real idea — scaling guidance by the ratio of current reward to a short history is simple and not in the cited guided-GRPO baselines. The systematic ablation of guidance ratio versus length is also useful, and the finding that naive fixed guidance fails while partial guidance helps is a concrete, reproducible insight. Credit where due: the experiments cover multiple model families and two task domains, and the code and models are released.\n\nThe soft spots are real but not fatal. The biggest is the α selection: Tables 8 and 9 pick the per-model, per-task guidance ratio by highest accuracy on the same benchmarks (MATH500, HumanEval, LiveCodeBench) that later appear in the headline Tables 3–5. That means the reported gains over vanilla GRPO are conditional on having tuned α to the test set. The paper never shows an untuned, α-default version still beats GRPO, so the \"adaptive guidance helps\" conclusion is confounded with \"the right fixed ratio helps.\" This is the load-bearing issue, and the stress-test note is correct.\n\nSecond, the claim that G2RPO-A \"consistently boosts accuracy\" is contradicted by the paper's own tables: Qwen3-0.6B on MATH500 drops from 54.26 (GRPO) to 51.77 (G2RPO-A), Qwen3-8B on HumanEval drops from 81.48 to 80.33, and Qwen3-1.7B on LiveCodeBench drops from 53.14 to 51.96. The ablation tables also show cases where fixed guidance beats G2RPO-A (e.g., Qwen3-1.7B on MATH500 with ℓ=2048). With single-seed runs and no variance estimates, those reversals could be noise, but they undercut the confident prose.\n\nThird, the paper cites Nath et al. and Park et al. as concurrent guided-GRPO work, but doesn't compare against Nath's \"adaptive guidance\" method. That's the one missing baseline that matters most. The paper also acknowledges that Eq. (5) breaks when batch difficulty varies, and patches it with curriculum ordering — fair, but it means the controller's premise (reward ratio ≈ learning progress) is untested in non-curriculum settings.\n\nBottom line: this is a worthwhile empirical contribution that deserves serious peer review, but it needs held-out α selection, multiple seeds, and a more honest characterization of when the method helps. I'd send it to review, expect a major revision, and would cite it as related work if I were working on small-model RLVR.","headline":"A genuinely new adaptive guidance controller with a solid ablation study, but test-set-tuned α and overclaimed consistency make the central comparison weaker than the paper admits.","tokens_in":15052,"tokens_out":3013,"would_cite":true,"duration_ms":29605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"G2RPO-A claims that adaptively shortening and lengthening injected reasoning guidance as rewards rise and fall lets small language models beat vanilla GRPO on math and code benchmarks.","keywords":["adaptive guidance","GRPO","reinforcement learning with verifiable rewards","small language models","curriculum learning","mathematical reasoning","code generation"],"falsifier":"Train G2RPO-A on a fixed dataset with batches shuffled randomly instead of curriculum-ordered, and compare its guidance-length trajectory and final accuracy against vanilla GRPO: if batch-difficulty fluctuations dominate reward changes, the length will oscillate and the reported accuracy gains should shrink or reverse. A more direct check is to correlate the sign of $\\ell_{k+1}-\\ell_k$ with held-out accuracy measured periodically during training; if the correlation is near zero or negative, the controller is not tracking competence.","tokens_in":14000,"feed_emoji":"🧭","tokens_out":4073,"duration_ms":39934,"temperature":0.7,"pith_summary":"The paper tries to establish that small language models can get substantially more out of reinforcement learning with verifiable rewards if their roll-outs are seeded with ground-truth reasoning traces whose length is adjusted automatically during training. It first shows that naively adding fixed-length guidance does not help: the reward curve quickly matches vanilla GRPO and the advantage signal becomes too flat. It then proposes G2RPO-A, which shrinks guidance length when recent rewards rise and grows it when they fall, and pairs this controller with an easy-to-hard curriculum. Across Qwen3 0.6B, 1.7B, and 8B, DeepSeek-Math-7B, and DeepSeek-Coder-6.7B, the paper reports consistent accuracy gains over vanilla GRPO on MATH500, Minerva, GPQA, AIME, HumanEval, and LiveCodeBench. If correct, the result is a cheap, training-time lever for improving small-model reasoning without a larger base model or a critic network.","feed_headline":"Small models beat GRPO with self-tuning reasoning guidance","feed_subtitle":"Adaptive guidance length shrinks when rewards rise and grows when they fall, lifting math and code scores.","key_machinery":"The central mechanism is the adaptive guidance-length controller in Eq. (5), a one-line ratio update that compares the current average reward $r_k$ with the average over the previous $\\min(T,k)$ steps and scales the guidance length $\\ell_k$ accordingly. Guidance itself is a ground-truth reasoning trajectory prepended to the prompt so the small model continues from a correct line of thought; the guidance ratio $\\alpha$ decides what fraction of each GRPO group of size $G$ receives guidance while the rest generate freely. The controller is protected from batch-difficulty noise by a curriculum that orders samples into ascending difficulty tiers (for math: cn_contest, aops_forum, amc_aime, olympiads, olympiads_ref). Together these pieces convert reward movement into a difficulty dial for the next training batch.","core_discovery":"On the paper's own terms, G2RPO-A 'markedly surpasses vanilla GRPO on nearly every benchmark' and 'consistently boosts accuracy.' With a per-model guidance ratio $\\alpha$ and an initial guidance budget of 3,072 tokens, the controller updates guidance length as $\\ell_{k+1} = \\ell_k \\cdot \\frac{\\min(T,k)\\, r_k}{\\sum_{\\tau=1}^{\\min(T,k)} r_{k-\\tau}}$ with $T=2$, so rising average reward shortens guidance and falling reward lengthens it. Combined with a curriculum ordering of training samples into ascending difficulty tiers, this produces reported improvements such as Qwen3-1.7B MATH500 rising from 63.74 to 67.21, Qwen3-8B MATH500 rising from 79.49 to 82.08, Qwen3-1.7B HumanEval rising from 67.65 to 75.93, and similar gains on Minerva, GPQA, AIME24/25, and LiveCodeBench. The paper also claims that the optimal guidance ratio depends on task and model size, with code tasks and smaller models needing stronger guidance, and that keeping hard samples with adaptive guidance beats filtering them out.","pith_inferences":["The controller uses a ratio of rewards rather than absolute reward values, so it may transfer across datasets with different reward scales; a direct test would be to normalize rewards and check whether the guidance-length trajectory stays the same.","A natural extension the paper does not pursue is applying the same reward-ratio controller to other RLVR variants, such as step-wise advantage methods, where guidance length could adapt to the model's competence at the token level.","The hand-chosen curriculum tiers are a patch for the controller's sensitivity to batch difficulty; an online difficulty estimator could replace them and make the method fully autonomous.","Under noisy or group-composition-driven reward fluctuations, the length update will oscillate; clamping, momentum, or a dead-zone on $\\ell_k$ would be a robustness fix the paper does not explore."],"forward_implications":["Small models can profit from RLVR on hard, sparse-reward tasks by keeping hard examples and attaching adaptive guidance, rather than filtering them out as too easy or too hard.","Because the controller needs only the last few average-reward values, it adds almost no overhead to a GRPO-style trainer and can replace manual guidance-length schedules.","The optimal guidance ratio still varies by model and task, so a practitioner would tune $\\alpha$ but can leave guidance length self-adjusting.","Curriculum ordering helps both vanilla GRPO and G2RPO-A, suggesting that presentation order matters independently of the adaptive controller.","If the reported gains hold, the approach is a practical alternative to distillation or much larger base models for improving small-model math and code reasoning."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that G2RPO-A builds on, and the DeepSeek-Math-7B base model used in math experiments.","marker":"[Shao et al., 2024]"},{"why":"Provides the DeepSeek-R1 RLVR success story that motivates applying RLVR to smaller models, and is cited as the GRPO reference in related work.","marker":"[Guo et al., 2025]"},{"why":"Provides the Open-R1 math-220k corpus from which the math training set with ground-truth solution trajectories is constructed.","marker":"[Wang et al., 2024]"},{"why":"Supplies the s1K-1.1 dataset used in the guidance-ratio and guidance-length configuration ablations.","marker":"[Muennighoff et al., 2025b]"},{"why":"Supplies the Qwen3 base models that are the primary backbones for the main experiments.","marker":"[Yang et al., 2025a]"},{"why":"Represents the online difficulty-filtering baseline that the paper contrasts against by arguing hard examples should be kept rather than removed.","marker":"[Bae et al., 2025]"},{"why":"Reinforce-Rej is the baseline that discards easiest and hardest examples, which G2RPO-A is shown to outperform by retaining tough samples.","marker":"[Xiong et al., 2025]"},{"why":"ADARFT is the success-rate-based curriculum ordering the paper tests and finds uninformative for its difficulty tiers.","marker":"[Shi et al., 2025]"},{"why":"Provides the curriculum-reinforcement-learning motivation for the easy-to-hard training order used in G2RPO-A.","marker":"[Parashar et al., 2025]"},{"why":"Supplies the HumanEval benchmark used to evaluate code-generation performance.","marker":"[Chen et al., 2021]"}],"fun_headline_variants":["Adaptive guidance raises small model math and code scores","Self-tuning reasoning guidance beats vanilla GRPO for small LLMs","G2RPO-A: length adjusts to reward, lifting SLM benchmarks","Adaptive guidance ratio improves SLM reasoning beyond GRPO","Reward-driven guidance length boosts small model math and code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a rise or fall in the current average reward is a faithful signal of the model's learning state, so shrinking guidance when rewards rise and growing it when they fall always tracks true competence; the paper itself concedes that when consecutive batches differ markedly in difficulty, reward changes no longer reflect learning progress, and it patches this with a hand-chosen curriculum order.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive guidance raises small model math and code scores","Self-tuning reasoning guidance beats vanilla GRPO for small LLMs","G2RPO-A: length adjusts to reward, lifting SLM benchmarks","Adaptive guidance ratio improves SLM reasoning beyond GRPO","Reward-driven guidance length boosts small model math and code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1445,"prompt_tokens":961,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":577,"tokens_out":484,"duration_ms":5178,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:17:02.110899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train G2RPO-A on a fixed dataset with batches shuffled randomly instead of curriculum-ordered, and compare its guidance-length trajectory and final accuracy against vanilla GRPO: if batch-difficulty fluctuations dominate reward changes, the length will oscillate and the reported accuracy gains should shrink or reverse. A more direct check is to correlate the sign of $\\ell_{k+1}-\\ell_k$ with held-out accuracy measured periodically during training; if the correlation is near zero or negative, the controller is not tracking competence.","supporting_citations":[],"review_version":2}