{"id":"3d414f42-2e6c-444d-8590-a4cdfa4b63fd","arxiv_id":"2505.12929","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Low-probability tokens dominate GRPO gradient updates; reweighting advantages by token probability or isolating low-probability tokens improves reasoning benchmarks.","lead":"This paper shows that during reinforcement learning, rare low-probability tokens can produce outsized gradients that crowd out learning for confident tokens, and proposes two simple fixes. A smart generalist might read it because the fixes are cheap and reportedly improve reasoning accuracy by up to 46% on logic puzzles.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dominance claim rests on per-token gradient norms; aggregate update contribution and |w| variation are unmeasured, so low-probability over-dominance is not established.","rationale":"The reader's conditional verdict is appropriate, and the weakest spot they identified is the same one I regard as load-bearing: the paper's mechanism requires that the per-token bound in Eq. (3) translates into aggregate gradient dominance, which depends on both the magnitude of w and on how token-gradient vectors combine. I add that the proposed fixes target only the advantage term, not the KL contribution inside w, so even the empirical gains do not cleanly confirm the stated mechanism unless the KL term is shown to be small. The paper does have real independent support: the released implementation, the consistent improvements across GRPO and REINFORCE++, and the reversed-order ablation in Figure 6(b) all suggest the effect is not purely random. But those results do not settle whether the mechanism is (1−π) dominance versus w-driven effects or simply a beneficial rebalancing of the optimization step. The proposed aggregate-gradient accounting is a single, feasible experiment that would settle the central claim. If it confirms that the low-probability group's net parameter-space gradient is the largest, the paper's interpretation is supported; if not, the theoretical explanation needs revision even if the heuristics remain useful. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":20332,"tokens_out":13326,"duration_ms":147955,"concrete_test":"Using the released code on Qwen2.5-3B-Instruct for one K&K GRPO step, log w_{i,t} and per-token gradient contributions from Eq. (8). Then compute, per probability quartile: (i) mean/median |w|; (ii) the L2 norm of the parameter gradient obtained by backpropagating Eq. (1) with only that quartile's token contributions active, masking all other tokens' policy and KL terms. If |w| varies substantially across quartiles, or if the low-probability quartile's aggregate gradient norm is not larger than the high-probability quartile's after controlling for token count, then Eq. (3) does not establish over-dominance. Report both group-wise |w| and aggregate gradient norms under the same masking used in Figure 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 4.2 bounds the norm of a single token's activation gradient by |w_{i,t}|(1−π). The paper then asserts w is approximately A and that low-probability tokens 'dominate model updates'. Two unverified joins break this chain. First, w_{i,t} = (πθ/πold) A·I + β(πref/πθ − 1) is not constant; for tokens whose probability has collapsed, β(πref/πθ − 1) can be large, so the observed gradient-size ordering may come from w rather than from (1−π). The proposed methods reweight or mask A while leaving the KL term in Eq. (2) active, so improved results would not confirm the proposed mechanism if w is the driver. Second, 'dominance' refers to the norm of the sum of hundreds of thousands of token gradients, not to per-token norms. Per-token norms can be large while the group sum cancels; Figure 1(d) reports only per-token statistics, and Figure 1(e,f) merely shows that updating low-probability tokens changes high-probability token probabilities, which any parameter update would do. Thus the central claim that high-probability token learning is suppressed by low-probability gradients is currently an inference, not a demonstrated property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that during GRPO training of LLMs, low-probability tokens produce disproportionately large gradient magnitudes, dominate the aggregate model update, and consequently suppress learning of high-probability tokens. It derives Proposition 4.2, bounding the per-token activation gradient norm by |w_{i,t}|(1 - pi_theta(o_{i,t})), and proposes two mitigation methods: Advantage Reweighting, which scales advantages by alpha*pi + (1 - alpha), and Low-Probability Token Isolation (Lopti), which updates low- and high-probability tokens in separate steps. Experimental results on K&K Logic Puzzles and math-related datasets show consistent improvements over naive GRPO, with up to a 46.2% relative gain on K&K Logic Puzzles, plus supporting ablations and a REINFORCE++ extension.","tokens_in":20611,"tokens_out":5685,"duration_ms":63712,"significance":"If the central claim is established, the paper offers a novel and practically useful diagnosis of an update bias in RL training for LLMs, well beyond the previously studied response-length bias. The strengths are the concise gradient derivation, the simplicity of the proposed methods, the public implementation, the coverage of multiple base models and datasets, and the informative reversed-order ablation in Figure 6(b). The main risk is that the paper's headline conclusion, that low-probability tokens dominate model updates, is currently an inference from per-token gradient bounds rather than a demonstrated property of the aggregate update; the empirical methods improve performance, but the measurements needed to tie those improvements to the proposed mechanism are missing.","major_comments":[{"comment":"The step from Proposition 4.2 to the 'dominance' claim treats |w_{i,t}| as approximately constant in Eq. (3), but this is not established and is in fact questionable. From Eq. (8), w_{i,t} = (pi_theta/pi_old) * A_hat * I + beta*(pi_ref/pi_theta - 1); the KL term grows as pi_theta becomes small, and for tokens whose negative advantage is clipped (I = 0), w is exactly that KL term and not A_hat. The sentence after Eq. (3) that w is 'approximately equal to A_hat in most cases' therefore needs quantitative support. A direct test would be to report mean/median |w_{i,t}| and the per-token GRPO gradient norm per probability quartile, with and without the KL contribution; if the norm ordering disappears once |w| is controlled, Proposition 4.2 does not explain the observed dominance. Since both proposed methods modify only the advantage part of w while leaving the KL term in Eq. (1) active, improved downstream performance cannot by itself confirm the proposed mechanism.","section":"§4.1, Eq. (3) and Eq. (8)"},{"comment":"The inference from per-token gradient norms to dominance of the aggregate model update is missing a load-bearing step. Proposition 4.2 bounds the norm of a single token's activation gradient, but an RL update is the sum of hundreds of thousands of token gradients; a group can have large per-token norms and yet a small or cancelling sum, while a group with small per-token norms can align constructively. Figure 1(d) reports only per-token statistics, and Figures 1(e) and 1(f) show only that updating low-probability tokens changes the probabilities of high-probability tokens, which any parameter update to a shared network would do. The paper should measure, for each probability quartile, the norm (and ideally the cosine similarity) of the summed token gradients used in an actual update, and show that the low-probability group's sum is large relative to the high-probability group's sum. This measurement is feasible and would directly support the term 'dominate model updates'.","section":"§4.1, Fig. 1(d)-(f)"},{"comment":"There is also an unaddressed gap between activation gradients and parameter gradients. Proposition 4.2 bounds delta_ell(o_{i,t}), the gradient with respect to an intermediate activation, while Figure 1(d) and the proposed methods concern parameter updates. The parameter-gradient norm contains an additional factor from the forward activations, so the bound in Eq. (3) does not by itself imply that parameter gradients follow the same (1 - pi) ordering. The authors should either extend the derivation to parameter gradients under a bounded-activation assumption or report parameter-gradient norms per probability quartile, which is the quantity that actually determines the update.","section":"§4.1, Eq. (3) vs. Fig. 1(d)"}],"minor_comments":[{"comment":"In Section 3, GRPO is attributed to reference [10] (Open-Reasoner-Zero); the correct source for GRPO is DeepSeekMath, reference [8]. The same citation issue appears near Eq. (1).","section":"§3 and References"},{"comment":"The table header contains a typo: 'Patially Correct' should be 'Partially Correct'.","section":"Table 4"},{"comment":"The sentence 'the estimated advantage is set to be consistence within each responses' should be reworded, e.g., 'the estimated advantage is set to be consistent within each response'.","section":"§3"},{"comment":"The hyperparameters alpha and eta are tuned on the same K&K and math datasets for which the headline results are reported, with recommended ranges derived from those datasets. Reporting results under a fixed default setting on a genuinely held-out task would make the empirical claims more robust.","section":"§5.1–5.3, Appendix B"},{"comment":"The model name 'Qwen2.5-7B-Instruct-1M' appears in the main text and tables without explanation of the '1M' variant; please clarify what this checkpoint is and why it differs from the standard Qwen2.5-7B-Instruct.","section":"Figure 4 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the proposed methods are simple and apparently effective. My main concern is that the central mechanistic claim, low-probability token over-dominance, rests on two unverified joins: the constancy/control of |w| and the passage from per-token norms to aggregate update dominance. Both are measurable, so I would encourage the editor to invite a revision that adds the suggested gradient-sum and |w| diagnostics. The reference attribution for GRPO should also be corrected before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a practical paper about a real nuisance in GRPO training: low-probability tokens in the sampled responses punch above their weight in the per-token gradient norm. Second, the paper's headline claim — that this 'over-dominance' suppresses learning of high-probability tokens — is asserted, not demonstrated; the theory only bounds a single token's activation gradient, and the weight w_{i,t} in front of (1−π) is not constant. That gap does not sink the methods, but it does change what the paper has actually shown.\n\nWhat's new: the observation that the softmax cross-entropy gradient for the selected token is (1−π) times the per-token weight is standard, but pointing out its consequence for RL updates and proposing two simple fixes is new. Advantage Reweighting and Lopti are simple, cheap (Lopti doubles update time, as the authors honestly note in Appendix E), and they give consistent gains on K&K logic puzzles and two math setups, sometimes large (0.39→0.57 avg on 3B, and REINFORCE++ 0.56→0.79 on 7B). That is enough to be useful to anyone doing RL fine-tuning.\n\nThe soft spots, in proportion. The theoretical Proposition 4.2 is correct but standard. The move from 'per-token gradient norm scales with 1−π' to 'low-prob tokens dominate the update' needs two things the paper doesn't supply: (i) an argument or measurement that |w_{i,t}| doesn't vary in a way that destroys the ordering — the KL term β(πref/πθ−1) can be large when π gets small, so the gradient ordering may be coming from w, not (1−π); and (ii) a direct check of the aggregate gradient contribution by probability group, since per-token norms can cancel in the sum. Figure 1(d) is per-token, and 1(e,f) only show that updating low-prob tokens changes high-prob token probabilities, which any parameter update would do. The ablation with reversed Lopti order collapsing is suggestive but not a mechanism proof. The empirical results also lack error bars and rely on a single seed; hyperparameters α, η are tuned on the same datasets, though the improvements look consistent across benchmarks.\n\nWho it's for: people building GRPO/RL pipelines who want a cheap, likely-helpful tweak. The methods deserve a serious referee and probably a revision, not a desk reject.","headline":"A practical, cheap fix for a real GRPO nuisance, but the 'low-probability token dominance' story is a plausible inference, not a proven mechanism.","tokens_in":21143,"tokens_out":3165,"would_cite":true,"duration_ms":30631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"During GRPO training, low-probability tokens generate disproportionately large gradients and dominate model updates, suppressing high-probability tokens; two methods—Advantage Reweighting and Lopti—correct this imbalance and improve…","keywords":["reinforcement learning","GRPO","token probability","gradient norms","low-probability tokens","advantage reweighting","LLM reasoning","policy gradient"],"falsifier":"On one of the paper's training setups, compute the full-batch GRPO gradient separately for tokens in the top and bottom probability quartiles, with the KL coefficient set to zero so the weight $w_{i,t}$ no longer depends on $\\pi_\\theta$; if the summed high-probability gradient norm is comparable to or larger than the low-probability one, or if per-token norm does not scale with $(1-\\pi)$, the dominance claim fails. A cheaper check is to compare how often positive tokens in each quartile increase in probability under Advantage Reweighting versus naive GRPO.","tokens_in":20129,"feed_emoji":"🤖","tokens_out":11053,"duration_ms":96154,"temperature":0.7,"pith_summary":"This paper argues that a hidden bias in RL for LLMs comes from token probability itself: the less probable a token is, the larger the gradient it produces, so low-probability tokens dominate each update and drown out high-probability tokens whose gradients are needed for good reasoning. The authors derive a per-token bound showing the gradient norm with respect to any layer's activations scales like $(1-\\pi)$, the complement of the token's probability, times a roughly constant weight. They then propose two fixes: Advantage Reweighting, which scales down advantages of low-probability tokens, and Low-Probability Token Isolation (Lopti), which updates low- and high-probability tokens in separate passes, low first. In experiments on K&K logic puzzles, combining the two raises GRPO-trained Qwen2.5-3B average accuracy from 0.39 to 0.57 (a 46.2% relative gain), with consistent gains on math datasets and under REINFORCE++. The paper's central message is that balancing token-level gradients, not just response-level rewards, is a cheap and effective lever in RL post-training.","feed_headline":"Reweighting low-probability tokens lifts RL reasoning up to 46%","feed_subtitle":"A gradient bound shows rare tokens crush high-probability signals; two cheap fixes restore balance in GRPO and beyond.","key_machinery":"The load-bearing identity is the logits-gradient factorization $\\delta_L(o_{i,t}) = w_{i,t}(I(o_{i,t}) - p(o_{i,t}))$, where $I$ is the one-hot vector at the sampled token and $p$ is the softmax distribution over the vocabulary; its norm lies between $\\sqrt{N/(N-1)}(1-\\pi)$ and $\\sqrt{2}(1-\\pi)$, which is the source of the $(1-\\pi)$ scaling. Backpropagation through layers with bounded Jacobian singular values preserves this scaling at every layer. The GRPO objective's token weight $w_{i,t} = (\\pi_\\theta/\\pi_{\\text{old}})\\,\\hat{A}_i\\,I_{\\text{trust}} + \\beta(\\pi_{\\text{ref}}/\\pi_\\theta - 1)$ is what the two proposed methods modify: Advantage Reweighting multiplies $\\hat{A}_i$ by $\\alpha\\pi_\\theta + (1-\\alpha)$, and Lopti masks the advantage so low- and high-probability tokens are updated in separate passes.","core_discovery":"The paper's central claim is that in GRPO training, the per-token gradient norm is essentially proportional to $1-\\pi_\\theta(o_{i,t})$, so tokens the model assigns low probability generate disproportionately large gradients. Proposition 4.2 states that under bounded-Jacobian assumptions, for any layer $\\ell$, $\\prod_{j=\\ell+1}^{L} c_j\\,|w_{i,t}|\\,\\sqrt{N/(N-1)}\\,(1-\\pi_\\theta(o_{i,t})) \\le \\|\\delta_\\ell(o_{i,t})\\| \\le \\prod_{j=\\ell+1}^{L} d_j\\,|w_{i,t}|\\,\\sqrt{2}\\,(1-\\pi_\\theta(o_{i,t}))$, where $w_{i,t}$ is the GRPO token weight from Eq. (2). Since updates average over hundreds of thousands of tokens, the large per-token norms of low-probability tokens dominate the summed gradient, and high-probability tokens—whose updates are more often in the correct direction—are suppressed. The paper supports this with probability-quartile experiments showing low-probability groups produce larger gradient norms and that updating only the lowest quartile reproduces the full-update probability shift. To counter the imbalance it introduces Advantage Reweighting (Eq. 4, replacing $\\hat{A}$ with $[\\alpha\\pi + (1-\\alpha)]\\hat{A}$) and Lopti (splitting tokens at threshold $\\eta$ and updating low-probability tokens first), reporting up to 46.2% relative improvement on K&K Logic Puzzles and gains on math benchmarks and REINFORCE++.","pith_inferences":["If the $(1-\\pi)$ scaling is the mechanism, the imbalance should worsen as training sharpens the policy: the highest-probability tokens become even quieter while low-probability tokens retain large gradients, so the reweighting gain should grow with training length and with stronger KL penalties—this is testable from the paper's own checkpoints.","The KL term in $w_{i,t}$ also grows as $\\pi_\\theta$ shrinks ($\\beta(\\pi_{\\text{ref}}/\\pi_\\theta - 1)$), so a sharper test of the paper's story would set $\\beta=0$ or freeze $\\pi_{\\text{ref}}$ to isolate whether the dominance comes from the softmax geometry or from the weight's own probability dependence.","Because the bound is derived for any layer activation, the same imbalance should appear in other policy-gradient variants and even in preference-optimization losses with a softmax head; applying Advantage Reweighting to those losses is a natural extension the paper does not pursue.","The probability-quartile gradient ratio suggests a practical monitoring metric: track each quartile's contribution to total gradient norm during RL and use it to schedule $\\alpha$ or $\\eta$, turning a static hyperparameter into an adaptive one."],"forward_implications":["Downweighting low-probability tokens by a factor linear in their probability (Advantage Reweighting) improves GRPO-trained accuracy on hard logic puzzles, with the largest gains on the most difficult 6- and 7-person puzzles where positive samples are scarce.","Updating low-probability tokens before high-probability tokens (Lopti) is the effective order; reversing the order makes training collapse after the fourth epoch, so the ordering itself is doing real work.","High-probability tokens cannot simply be discarded: masking them degrades baseline GRPO, so the correct intervention is to attenuate low-probability interference rather than remove high-probability gradients.","The imbalance is not specific to GRPO: the same reweighting improves REINFORCE++ on K&K puzzles (for example, average accuracy on Qwen2.5-3B rises from 0.23 to 0.34), suggesting policy-gradient LLM training broadly is affected.","On math benchmarks the two fixes each help but do not stack; the paper recommends using them individually there, implying the benefit is real but not additive in all settings."],"supporting_citations":[{"why":"Supplies the GRPO algorithm whose token-level updates this paper analyzes.","marker":"[8]"},{"why":"Establishes the simple rule-based-reward RL recipe that motivates studying GRPO's update bias.","marker":"[2]"},{"why":"Provides the K&K logic puzzle training set and rule-based reward function used in the main experiments.","marker":"[38]"},{"why":"Aggregated the K&K logic puzzle benchmark the paper evaluates on.","marker":"[45]"},{"why":"Documents response-length update bias in GRPO and supplies the 'clip higher' stabilization used in training.","marker":"[9]"},{"why":"Compares GRPO with PPO and finds update bias from incorrect responses; supports the paper's claim that positive samples matter more.","marker":"[11]"},{"why":"Serves as the open-source implementation base for the GRPO training setup.","marker":"[44]"},{"why":"Provides the REINFORCE++ baseline used to show the token-dominance fix transfers beyond GRPO.","marker":"[46]"}],"fun_headline_variants":["Tame rare tokens, gain 46% on RL reasoning","Low-prob tokens hijack gradients; reweight to fix","Cut low-prob token noise for 46% better reasoning","Rare tokens crush RL updates; new fix lifts 46%","Reweight rare token gradients, boost RL 46%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that low-probability tokens dominate rests on treating the token weight $w_{i,t}$ as roughly constant across probability groups; in fact $w$ includes $\\beta(\\pi_{\\text{ref}}/\\pi_\\theta - 1)$, which grows as $\\pi_\\theta$ shrinks, and the paper does not show $|w|$ is similar across groups, nor that per-token gradient norms survive summation without cancellation across the hundreds of thousands of tokens in each update.","fun_headline_variants_meta":{"raw":{"variants":["Tame rare tokens, gain 46% on RL reasoning","Low-prob tokens hijack gradients; reweight to fix","Cut low-prob token noise for 46% better reasoning","Rare tokens crush RL updates; new fix lifts 46%","Reweight rare token gradients, boost RL 46%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1778,"prompt_tokens":1098,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":594}},"tokens_in":714,"tokens_out":680,"duration_ms":7351,"temperature":1.0,"reasoning_tokens":594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:22:43.596899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On one of the paper's training setups, compute the full-batch GRPO gradient separately for tokens in the top and bottom probability quartiles, with the KL coefficient set to zero so the weight $w_{i,t}$ no longer depends on $\\pi_\\theta$; if the summed high-probability gradient norm is comparable to or larger than the low-probability one, or if per-token norm does not scale with $(1-\\pi)$, the dominance claim fails. A cheaper check is to compare how often positive tokens in each quartile increase in probability under Advantage Reweighting versus naive GRPO.","supporting_citations":[{"cited_title":"Hybridflow: A flexible and efficient RLHF framework","cited_arxiv_id":null,"evidence_quote":"Serves as the open-source implementation base for the GRPO training setup."}],"review_version":1}