{"id":"6ccfe232-3f14-4db4-8d72-f6f8dc7de11e","arxiv_id":"2607.07976","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TACO soft-suppresses positive GRPO credit on high tail-risk tokens (surprisal above local entropy) and consistently beats GRPO-style baselines on three LLMs and eight reasoning benchmarks while stabilizing long training.","lead":"Critic-free RL for LLM reasoning (GRPO-style) broadcasts one advantage to every token, so rare, contextually wrong tokens in a correct answer still get reinforced. TACO down-weights those positive updates using a cheap surprisal-minus-entropy tail-risk score and improves accuracy and long-horizon stability on math and science benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing concern is whether r_tail truly separates implausible from useful rare tokens, or whether gains mainly come from generic positive-advantage damping of low-p tokens.","rationale":"The reader correctly identifies the weakest assumption: that the fixed-α surprisal-minus-entropy proxy reliably distinguishes implausible tail tokens from useful exploration so that positive-only soft suppression improves the policy for the right reason. That assumption is load-bearing for both the diagnosis (Positive-Credit Contamination as the operative failure mode) and the method claim (TACO as a context-aware fix rather than another low-p stabilizer). The synthetic MDP and qualitative cases are supportive but not decisive; the main tables lack multi-seed error bars and are Qwen-centric, which the reader already notes. My concrete test directly probes the missing independent validation and the necessity of the entropy term. Because the empirical gains and long-horizon stability results still stand as reported, and code is linked, I do not move the verdict below CONDITIONAL; addressing the token-level validation and rarity-only ablation would be the natural path toward ACCEPT. No stronger internal inconsistency or circularity is present.","tokens_in":16807,"tokens_out":675,"duration_ms":6723,"concrete_test":"On a held-out set of rewarded training traces, collect human or process-verifier labels of locally erroneous vs. useful rare tokens; report precision/recall of r_tail > 0 and of the full weight schedule, and ablate Eq. (2) by replacing it with rarity-only (e.g., −log p + c) and entropy-only variants. If the entropy-aware score does not improve token-level reliability metrics and the rarity-only ablation matches Table 1 average gains within ~0.5 points, the separation claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that Eq. (2) (r_tail = −log p − H + log α) plus positive-only soft weighting (Eqs. 3–4) specifically suppress Positive-Credit Contamination rather than merely reweighting low-probability tokens under positive advantages. The paper motivates the score via nucleus-sampling literature and a synthetic MDP (Fig. 3) where optimal vs. tail actions are oracle-labeled, and shows qualitative cases (Figs. 6, 10, 11) where noisy fragments receive lower weight. However, there is no independent token-level validation (human or process labels) that high r_tail tokens are disproportionately the ones that are contextually erroneous inside rewarded traces, nor an ablation that isolates the entropy term from pure rarity. Diagnostics (Fig. 9) show only ~1.9% of positive-advantage tokens are downweighted on average, so small mislabeling of useful rare patterns could still matter. If the entropy adjustment is not doing the claimed separation work, the gains over GRPO / Adv. Reweighting / STAPO could be misattributed to a more generic low-p positive-credit dampener, weakening the causal link from the diagnosed failure mode to TACO’s mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper identifies Positive-Credit Contamination as a failure mode of GRPO-style critic-free RLVR: sequence-level positive advantages are broadcast uniformly, so locally implausible low-probability tokens inside rewarded completions receive the same positive credit as reliable tokens. It proposes TACO, which scores each token with a tail-risk measure r_tail = -log p - H + log α (Eq. 2) and softly down-weights only positive advantages for high-risk tokens (Eqs. 3–4), leaving negative advantages unchanged. Empirically, TACO is evaluated on three base models (Qwen3-1.7B/4B-Base, Qwen2.5-Math-7B) and eight benchmarks (six math, two OOD science), with comparisons to GRPO (clip-higher), advantage reweighting, and STAPO. Reported gains are consistent on average accuracy, with additional evidence from training dynamics, a 600-step long-horizon run where GRPO plateaus while TACO continues to improve, qualitative token-suppression cases, a controlled synthetic MDP isolating broadcast vs. oracle credit, and released code.","tokens_in":17138,"tokens_out":1837,"duration_ms":24763,"significance":"If the diagnosed failure mode and the proposed calibration are correct, the work is a useful, low-overhead contribution to reasoning-oriented RLVR: it names a concrete credit-assignment pathology of broadcast advantages, gives a simple generation-time fix that does not require critics or external process labels, and shows multi-scale, multi-benchmark gains plus improved long-horizon stability. Strengths that should be credited include the synthetic MDP that isolates broadcast contamination under controlled optimal/tail actions (Fig. 3), the multi-model evaluation with OOD science sets (Table 1), the extended-training comparison (Fig. 5), qualitative suppression cases (Figs. 6, 10, 11), training diagnostics showing sparse intervention (~1.9% of positive-advantage tokens; Fig. 9), and public source code. The practical significance is real even if the method is viewed as a carefully designed positive-only low-probability dampener rather than a fully validated semantic reliability filter.","major_comments":[{"comment":"The central mechanistic claim is that Eq. (2) separates contextually implausible tail tokens from useful rare exploration via the entropy adjustment, so that Eqs. (3)–(4) specifically mitigate Positive-Credit Contamination rather than generically damping low-p tokens under positive advantages. The manuscript motivates this with nucleus-sampling literature, the synthetic MDP (Fig. 3), and qualitative cases (Figs. 6, 10, 11), but does not ablate the entropy term: a pure-rarity baseline (e.g., r = -log p + c, or positive-only reweighting of low-p tokens without H) is missing. Without that ablation, gains over GRPO / Adv. Reweighting / STAPO (Table 1) cannot be confidently attributed to the claimed context-aware separation. Please add this ablation on at least one model and report both accuracy and which tokens are suppressed.","section":null},{"comment":"Relatedly, there is no independent token-level check that high r_tail tokens inside rewarded traces are disproportionately the contextually erroneous ones. Fig. 9 shows only ~1.9% of positive-advantage tokens are downweighted on average, so mislabeling of useful rare patterns could still matter. Human or process-level labels on a sample of rewarded traces (or a proxy such as step-level verifier disagreement / self-consistency under local edits) would substantially strengthen the causal link from the diagnosed failure mode to TACO. If such labels are infeasible, at least quantify false-positive suppression rates on known useful rare tokens (e.g., uncommon but correct mathematical symbols) versus clear noise fragments.","section":null},{"comment":"Section 5.2 and Table 1 frame TACO as improving by calibrating implausible positive credit 'beyond merely stabilizing' low-p updates as in STAPO and Adv. Reweighting. Those baselines address gradient dominance / spurious large updates, not the same positive-only contamination objective. To make the comparison load-bearing for the paper's claim, either (i) include a positive-only low-p dampener matched to TACO's intervention rate, or (ii) analyze whether TACO's suppressed tokens differ systematically from those affected by STAPO/Adv. Reweighting (overlap, entropy regime, position in the trace). Currently the superiority claim is empirical but the mechanistic contrast is under-specified.","section":null},{"comment":"The synthetic MDP (Sec. 3.3, Fig. 3, App. B) usefully shows that broadcast advantages underperform oracle step credit as H grows, n_opt shrinks, and G shrinks. However, the MDP's 'tail' actions are oracle-labeled suboptimal actions with reward 0, whereas LLM implausible tokens are defined via policy probability and local entropy without outcome labels. Please clarify what this experiment does and does not establish: it supports that broadcast contamination can hurt, but it does not validate that r_tail recovers the oracle unreliable set under LLM-like distributions. A short bridging experiment (e.g., applying r_tail-style scoring inside the MDP and measuring recovery of suboptimal actions) would tighten this link.","section":null}],"minor_comments":[{"comment":"Eq. (2): the role of log α is described as making identification more aggressive for larger α, but the sign convention (adding log α so that r_tail > 0 marks risk) would be clearer with a short derivation or a one-line interpretation of the threshold p < α e^{-H}.","section":null},{"comment":"Table 2 reports sensitivity for α ∈ {0.01, 0.005} and λ ∈ {0.6, 0.9}; the text mentions clear early degradation at α = 0.1 but does not tabulate it. Including that failed setting would help readers see the failure mode of over-aggressive tail identification.","section":null},{"comment":"Figure 1 caption and main text refer to 'five representative benchmarks' while the abstract and Sec. 5 claim eight; align the wording so the figure is clearly a subset of Table 1.","section":null},{"comment":"Notation: ℒ_{i,t} is introduced as the PPO-style clipped surrogate in Sec. 3.2 with a footnote to DeepSeek-R1; a self-contained formula (or explicit pointer to the exact clipped objective used in verl) would improve reproducibility for readers not carrying that report.","section":null},{"comment":"In Fig. 6 and related cases, red/blue highlighting is informative; stating the exact mapping from w_{i,t} to color intensity (and whether weights are computed under π_θ_old or π_θ) would make the qualitative evidence easier to interpret.","section":null},{"comment":"Minor polish: 'i.os ales' and mixed-language fragments in Fig. 6 are intentional examples of noise, but a few prose typos elsewhere (e.g., spacing around TACO in early sections, 'V olume' in bibliography) should be cleaned in revision.","section":null},{"comment":"App. A: evaluation uses top-p=0.7, temperature 0.9 while training uses temperature 1.0, top-p=1.0. A brief note that this is intentional (and shared across methods) would preempt fairness concerns.","section":null}],"recommendation":"major_revision","confidential_remarks":"The empirical package is above average for this line of work (multi-scale models, OOD sets, long-horizon run, code). My major_revision recommendation is driven by the gap between the strong causal framing (Positive-Credit Contamination cured by entropy-aware tail risk) and the missing ablations that would pin down that mechanism. If the authors deliver the entropy ablation and a clearer baseline contrast, this could become a solid accept at minor revision; if they reframe more modestly as positive-only low-p credit shaping with good empirical results, the bar for acceptance is lower. Scope fit for a serious ML/NLP venue is fine."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this paper names a concrete failure mode in GRPO-style RLVR—Positive-Credit Contamination, where implausible low-p tokens inside rewarded traces get the same positive advantage as good ones—and ships a near-zero-overhead soft fix (TACO) that consistently beats GRPO, advantage reweighting, and STAPO on three Qwen-scale models and eight math/science benchmarks, with better late-training behavior out to 600 steps.\n\nWhat is new is not “token reweighting exists,” but the specific framing plus the rule: r_tail = −log p − H + log α, then positive-only soft downweighting that leaves negative advantages alone and never zeros gradients. That is a usable distinction from pure rarity damping, gradient clipping (STAPO), or external process signals. The synthetic MDP isolating broadcast vs oracle credit is a good controlled argument that the pathology is real under long traces and sparse optimal actions. Training dynamics, qualitative cases, and the diagnostic that only ~1.9% of positive-advantage tokens are touched make the story coherent. Code is linked; writing is clear; citations cover the right neighbors without pretending the field is empty.\n\nSoft spots, in proportion: the load-bearing proxy (surprisal minus entropy) is motivated by nucleus sampling and the MDP, not by independent human/process labels of token reliability, and there is no clean ablation that isolates the entropy term from pure low-p positive damping. Main tables lack multi-seed error bars, and everything is Qwen-family. Those are real but ordinary for this venue class; they do not collapse the empirical claim that the method works. Hyperparameters α, λ are free and validated, which is fine.\n\nThis is for people who actually train critic-free reasoning RL and care about long-horizon stability. It deserves a serious referee, not a desk reject. I would engage with it, cite the failure-mode framing and the long-horizon plot, and push for the entropy ablation and error bars in revision.","headline":"Clean diagnosis of a real GRPO credit bug plus a cheap, usable fix that actually moves numbers and long-horizon stability.","tokens_in":17778,"tokens_out":506,"would_cite":true,"duration_ms":5500,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Critic-free RL for language models gives the same positive credit to bad low-probability tokens as to good ones; TACO recalibrates that credit so flawed local steps stop getting reinforced.","keywords":["reinforcement learning","large language models","credit assignment","GRPO","tail tokens","reasoning","policy optimization","Positive-Credit Contamination"],"falsifier":"Train the same model with TACO and with GRPO, then have independent process-level or human labels mark which tokens in rewarded trajectories are locally erroneous; if TACO’s high-risk tokens do not disproportionately match those erroneous tokens, or if forcing the same down-weighting on randomly chosen rare tokens produces equal gains, the central claim fails.","tokens_in":17685,"feed_emoji":"⚖️","tokens_out":742,"duration_ms":11886,"temperature":0.7,"pith_summary":"When language models are trained with group-relative, critic-free reinforcement learning, a single success signal for a whole answer is copied onto every token in that answer. The paper argues that this broadcast rule creates Positive-Credit Contamination: locally wrong or nonsensical low-probability tokens that happen to sit inside a rewarded answer receive the same positive update as the tokens that actually carried the reasoning. Over training those bad local habits get strengthened. The authors propose TACO, which scores each token’s risk of belonging to the unreliable tail of the next-token distribution using its probability and the local entropy of the context, then softly reduces only positive credit for high-risk tokens. Across three base models and eight reasoning benchmarks, this calibration beats standard GRPO-style training and related stabilizers, and it keeps improving under longer training where the baseline plateaus or declines. A reader who cares about reliable chain-of-thought training should care because the method attacks a concrete failure mode of the dominant cheap RL recipe without adding a critic or heavy external feedback.","feed_headline":"RL was rewarding bad tokens inside good answers","feed_subtitle":"TACO soft-suppresses positive credit on risky tail tokens and keeps gains rising where GRPO stalls.","key_machinery":"Tail-Aware Credit Calibration (TACO): a tail-risk score r_tail = −log p − H + log α that compares a token’s surprisal to the local entropy of the generation context, converted into a weight that multiplies only positive sequence advantages so high-risk tokens are softly down-weighted rather than zeroed out.","core_discovery":"The central discovery is that Positive-Credit Contamination is a real failure mode of GRPO-style reinforcement learning with verifiable rewards: implausible low-probability tail tokens inside otherwise correct trajectories receive identical positive sequence-level advantages, and the accumulation of those updates biases the policy toward flawed local continuations. TACO mitigates this by estimating a context-aware tail-risk score and using it to soft-suppress only positive token-level credit, preserving useful rare exploration while dampening incidental noise, with consistent gains on in-domain math and out-of-distribution science benchmarks and more stable long-horizon training.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Uniform RL credit boosts bad tail tokens in good paths","Implausible tokens get full advantage inside correct answers","Positive-credit contamination lets noise hijack verifiable rewards","TACO soft-suppresses positive credit on risky low-prob tokens","Sequence advantages reinforce flawed local tokens unchecked"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes that a simple score built from a token’s probability and the local entropy of its context can tell useful rare exploration apart from contextually bad tail tokens well enough that soft positive-only down-weighting improves the policy rather than just swapping one bias for another.","fun_headline_variants_meta":{"raw":{"variants":["Uniform RL credit boosts bad tail tokens in good paths","Implausible tokens get full advantage inside correct answers","Positive-credit contamination lets noise hijack verifiable rewards","TACO soft-suppresses positive credit on risky low-prob tokens","Sequence advantages reinforce flawed local tokens unchecked"]},"model":"grok-4.5","effort":"low","cost_usd":0.004298,"raw_usage":{"total_tokens":1339,"prompt_tokens":838,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":42980000,"prompt_tokens_details":{"text_tokens":838,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":421,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":838,"tokens_out":80,"duration_ms":4416,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T14:24:03.421134+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same model with TACO and with GRPO, then have independent process-level or human labels mark which tokens in rewarded trajectories are locally erroneous; if TACO’s high-risk tokens do not disproportionately match those erroneous tokens, or if forcing the same down-weighting on randomly chosen rare tokens produces equal gains, the central claim fails.","supporting_citations":[],"review_version":1}