{"id":"8f52fc2b-5886-4db3-8a09-fa11bb90b3c4","arxiv_id":"2608.13179","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CREST, a hierarchical credit assignment method, improves multi-turn tool-use agent training by combining turn-level advantages with entropy-gated self-teacher token modulation, outperforming RL and distillation baselines.","lead":"This paper introduces CREST, a training method that gives AI agents better feedback at each step of multi-step tasks by splitting rewards into per-turn parts and using the model itself to focus learning on important tokens. The result is stronger tool-use agents on benchmarks, with the largest gains on long sessions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verifier-bounded ceiling is asserted globally but proved only per-token; the appendix disclaims aggregate direction preservation, and the 8.4% bias bound is misstated as relative.","rationale":"The reader's verdict is CONDITIONAL, and I agree with the overall assessment, but I would locate the load-bearing weakness differently. The reader's weakest_assumption concerns per-turn reward alignment; that is a real scope limitation, but it is explicitly acknowledged in Section 4.1 and is a property of the benchmarks rather than a flaw in the argument. The more serious issue is internal: the proof appendix explicitly disclaims the aggregate implication that the abstract and conclusion assert. This is not a matter of wording, because the central contribution - 'teach the magnitude, not the direction' - is precisely the claim that the teacher cannot control update direction. If the appendix is right, then the teacher can reshape the aggregate gradient by changing which tokens dominate, so CREST is not verifier-bounded in the sense advertised; it is a reweighted policy gradient with a per-token sign constraint. The P2 misstatement in Section 3.4 compounds this: the stated relative 8.4% bound is not what is proved. Both issues are fixable by rewording and by reporting aggregate-gradient diagnostics, so the paper remains conditionally acceptable rather than rejected. I set verdict_should_be to CONDITIONAL and agreement_with_reader to partial: the reader flagged the overstatement in passing but did not make it the weakest assumption, while I see it as the primary load-bearing concern.","tokens_in":15671,"tokens_out":8348,"duration_ms":77627,"concrete_test":"On a logged CREST training run (BFCL V3 / Qwen3-4B), compute per step the teacher perturbation d_t = sum_t (phi_t - 1) g_t and the inter-turn-only gradient g_0 = sum_t g_t (with phi_t = 1), where g_t = A_turn[t] grad log pi(y_t). Report (i) the maximum over steps of ||d_t|| / ||g_0|| versus lambda*epsilon = 0.084, and (ii) the minimum cosine similarity between the full CREST aggregate gradient and g_0. If the ratio exceeds 0.084 or the cosine falls materially below 1 on any step, the Section 3.4 '8.4% bounded bias' and the abstract's 'direction determined by verifier' claims fail in the actual training regime, and the paper must be reframed as offering only token-wise sign preservation, not aggregate verifier-boundedness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central novelty is that CREST is 'verifier-bounded by construction' because the self-teacher only modulates magnitudes while verified rewards set every gradient direction (abstract, Section 3, Section 5). The formal support is Proposition A.1, but it proves only token-wise scalar-sign preservation (sign(A_t) = sign(A_turn[t])) because phi_t > 0. The appendix itself states the limitation: reweighting 'can still change how token gradients cancel in the aggregate, so it does not imply that aggregate gradient directions, stationary points, or a global performance ceiling are determined solely by the verifier' (Section A.1, repeated in Section A.4). The conclusion and abstract nevertheless claim the verifier-bounded ceiling is preserved. Additionally, Section 3.4 states P2 as ||Bias|| <= lambda*epsilon*||grad J_GRPO|| ~ 8.4%, but Appendix A.2 proves only an absolute bound ||sum_t (phi_t - 1) g_t|| <= lambda*epsilon*sum_t ||g_t||; the former does not follow from the latter because sum ||g_t|| can be much larger than ||sum g_t||. Thus the headline decoupling - teacher as magnitude-only, verifier as direction - is true only at the level of individual token scalar signs, not at the level of the parameter update that actually determines training. Since the title, motivation, and Section 5 all rest on this decoupling, the central claim overreaches what is proved. The empirical results may survive, but the theoretical guarantee that distinguishes CREST from a heuristic reweighting scheme is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CREST, a hierarchical credit-assignment method for training multi-turn, multi-step LLM tool-use agents. The method replaces trajectory-level advantages with per-turn group-relative advantages computed from verifier rewards, and multiplies each token's advantage by an entropy-gated, self-teacher-derived magnitude factor. The authors claim that this decomposition keeps gradient directions determined by the verifier while letting a privileged self-teacher modulate only per-token magnitudes, and they report experiments on BFCL V3 and WildToolBench with Qwen3-4B and Qwen3-8B, together with ablations and a formal appendix.","tokens_in":15931,"tokens_out":8071,"duration_ms":73529,"significance":"If the claims were fully supported, the paper would make a useful practical contribution: the turn-segmented advantage is a simple and plausible fix for inter-turn reward dilution, and the entropy-gated modulation is a sensible guard against gradient concentration collapse. The paper is clearly written and the appendix is unusually candid, explicitly stating that the formal guarantee is token-wise scalar-sign preservation and that the perturbation bound is absolute, not relative to the aggregate gradient. The two-level ablation is a helpful design analysis. However, the headline conceptual claim, that the teacher is restricted to magnitude modulation without sacrificing a verifier-bounded ceiling, is not established beyond the token-wise level, and the main empirical tables rest on single-seed runs with no variance estimates. The contribution is therefore currently overstated relative to the evidence, but the underlying approach remains plausible and the issues appear addressable in revision.","major_comments":[{"comment":"The main text's statement of property P2 is not supported by the proof. Section 3.4 states that ||Bias|| ≤ λε·||∇J_GRPO|| ≈ 8.4%, which is a relative bound on the aggregate GRPO gradient. Appendix A.2 proves only the absolute bound ||Σ_t (φ_t−1)g_t|| ≤ λε Σ_t ||g_t||. The latter does not imply the former because Σ_t ||g_t|| can be much larger than ||Σ_t g_t|| when token gradients partially cancel. The proof itself is correct, but the main-text formulation and the derived '8.4%' claim must be corrected, and all downstream statements that rely on the relative interpretation should be revised.","section":"§3.4 and Appendix A.2"},{"comment":"The central claim that CREST 'preserves the verifier-bounded ceiling' overreaches what is proved. Proposition A.1 establishes only that sign(A_t) = sign(A_turn[t]) for each token individually, because φ_t > 0. Appendix A.1 and A.4 explicitly state that reweighting can change how token gradients cancel in the aggregate and that the result does not imply preservation of aggregate gradient directions, stationary points, or a global performance ceiling. The abstract and Section 5 nevertheless assert that the verifier-bounded ceiling is preserved 'by construction' and that the teacher's role is reduced to magnitude modulation. This mismatch between the formal content and the framing is load-bearing because the paper's title and motivation rest on the verifier-bounded-ceiling claim. The authors should either prove an aggregate-level guarantee or reframe the claim as a token-wise anchor plus a heuristic, and adjust the abstract and conclusion accordingly.","section":"Abstract, §5, Appendix A.1, A.4"},{"comment":"The empirical evidence for the central 'consistently outperforms all baselines' claim is weakened by the single-training-seed protocol. Section 4.1 states that each method uses a single training seed and that the three evaluation decodes are not independent training replicates; no standard errors or multiple-seed results are reported. Several margins in Table 2 are small relative to what would be expected from training noise, for example Qwen3-8B WildToolBench Task Accuracy (CREST 52.34 vs MT-GRPO 49.61) and Qwen3-4B BFCL Base (CREST 67.0 vs MT-GRPO 63.0). Variance estimates or additional seeds are needed to support the strength of the empirical claim. Additionally, the Qwen3-4B CREST row in Table 2 reports an Average of 52.00, but the four split values (67, 48, 38, 60) have arithmetic mean 53.25, and the reported improvement +29.88 does not match the column values; because the abstract quotes 52.0%, this numerical inconsistency must be resolved.","section":"Table 2 and §4.1"}],"minor_comments":[{"comment":"The title contains a typo, 'Multi-T urn', which should read 'Multi-Turn'; the same issue appears in the first line of the full text.","section":"Title and §1"},{"comment":"The symbol 'Bias' in the P2 statement is used before it is defined; the authors should define it as the teacher-induced perturbation D = Σ_t (φ_t−1)g_t relative to the inter-turn-only gradient.","section":"§3.4"},{"comment":"Algorithm 1 and Section C.1 describe the self-teacher as conditioning on ground-truth tool-call results, but the construction of the privileged context h^T_t is not fully specified; a concrete description of how the teacher context is assembled for each turn would improve reproducibility.","section":"Appendix B / C.1"},{"comment":"The sentence 'we adopt the fixed teacher with per-token KL clipping (threshold 0.2) for the OPSD baseline' is clear, but it would help to state explicitly that this fixed-teacher variant is the one plotted as the OPSD curve in Figure 3(a), since the caption alone does not say so.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a real practical idea, and the appendix's honesty about the token-wise scope of the formal results is commendable. However, the main text overstates the formal guarantee, the P2 bound is misstated in a load-bearing way, and the empirical support is based on single-seed runs. I would want the authors to correct the formal-claim mismatch, fix the Table 2 arithmetic, and either add multi-seed results or substantially soften the consistency claims before this is publishable. The issues are fixable, but they are not merely editorial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The central idea here is genuinely useful: split per-token credit into a sign, set by the verifier, and a magnitude, modulated by an entropy-gated self-teacher. That decomposition is new, as far as I know, and it gives a clean explanation of why pure distillation collapses in this setting. The paper also does the right formal work for the narrow claim it can prove—token-wise sign preservation (P1), a perturbation bound (P2), and sign-consistent amplification (P3) are all proved correctly in the appendix, and the appendix is honest that these do not constrain aggregate gradient directions or a global performance ceiling. The ablation separating inter-turn and intra-turn contributions is informative, and the OPSD baseline is handled fairly with a fixed teacher and KL clipping.\n\nThe soft spots are real but not disqualifying. Most important, the abstract and conclusion sell the result as preserving 'the verifier-bounded ceiling' without qualification, while the appendix explicitly says reweighting can change how token gradients cancel in aggregate, so aggregate directions, stationary points, and a global ceiling are not determined solely by the verifier. The gap between headline and proof is the main thing a referee should push on. Relatedly, Section 3.4 states the P2 bound as relative (8.4% of the GRPO gradient norm), but the appendix proves only an absolute bound on the sum of per-token perturbations; the two are not the same, and the relative reading is unsupported.\n\nThe empirical base is thinner than the abstract suggests. Single training seed, no error bars, no code, method-specific training step counts without a stated early-stopping protocol, and a 'processed' BFCL split that uses 100 fixed IDs for training. The method also presupposes per-turn verifier rewards that are semantically aligned across rollouts; the authors acknowledge this, so it is not hidden, but it does limit the method to benchmarks with that structure. None of these are fatal—the gains are consistent across the ablations and both benchmarks—but they mean the numbers should be read as suggestive, not definitive.\n\nNet: the method is plausible and the formal token-wise claims are correct; the ceiling overreach and thin empirical reporting are addressable with code, multiple seeds, and reworded guarantees. I'd send this to a serious referee, but I'd expect heavy revision. If you work on multi-turn agent RL, it is worth a reading-group slot.","headline":"The verifier-bounded ceiling claim overreaches the token-wise proof, and the single-seed numbers need support, but the sign/magnitude decomposition is a fresh, buildable idea that deserves refereeing.","tokens_in":16515,"tokens_out":3094,"would_cite":true,"duration_ms":37102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CREST claims that a self-teacher should set update magnitudes while the verifier sets update directions, and that this split unlocks dense token-level credit assignment for multi-turn tool-use agents without losing the verifier-bounded…","keywords":["multi-turn agents","credit assignment","reinforcement learning","verifiable rewards","self-distillation","tool use","policy optimization","entropy gating"],"falsifier":"Run CREST on a multi-turn benchmark where only session-level success is observable, or where rollouts within a group have different turn counts and orders; if the method then no longer beats GRPO and MT-GRPO on the held-out splits, the turn-alignment assumption fails. A second check is to match the OPSD setup and measure the top-5% token gradient share: if CREST's share approaches OPSD's roughly 77% instead of the reported middle ground, the entropy gate is not preventing concentration collapse.","tokens_in":15430,"feed_emoji":"🧭","tokens_out":7553,"duration_ms":64347,"temperature":0.7,"pith_summary":"The paper tries to solve credit assignment for multi-turn, multi-step LLM agents, where a trajectory mixes successful and failed turns and a single trajectory-level reward mislabels tokens in failed turns. Its proposal, CREST, decomposes each token's training signal into a turn-level verified advantage, which says which turn earned the credit, and a token-level magnitude modulation from a privileged self-teacher, which says which tokens within the turn matter most. The central claim is that the teacher signal never flips the sign of any token's update, only scales it, so the verifier remains the sole source of direction. If true, this gives multi-turn agents dense token-level supervision without giving up the verifier-bounded performance ceiling that makes reinforcement learning with verifiable rewards reliable.","feed_headline":"Verifier sets the sign, teacher sets the size","feed_subtitle":"Two-level credit assignment lifts tool-agent accuracy on BFCL V3 and WildToolBench, especially long sessions.","key_machinery":"The central object is the factored per-token advantage $$A_t = $A^{{\\mathrm{turn}}$}_{[t]}\\,\\phi_t, \\qquad \\phi_t = 1 + \\$lambda^{{\\mathrm{eff}}$}_t\\,(w_t - 1),$$ where $A^{\\mathrm{turn}}_{[t]}$ is a group-relative normalized verified reward computed independently within each turn, and $\\phi_t$ is a positive multiplicative factor built from a teacher--student divergence $\\Delta_t$, clipped into $w_t$, then gated by a direction gate that suppresses teacher--verifier disagreement and an entropy gate that concentrates amplification on high-uncertainty content tokens. Because $\\phi_t > 0$ always, the scalar sign of every token's update is identical to the sign of the turn-segmented verified advantage; the teacher can only enlarge an update along a verifier-approved direction. The direction gate is the mechanism that keeps the method verifier-bounded rather than teacher-bounded, while the entropy gate prevents the gradient-concentration collapse observed in plain self-distillation.","core_discovery":"CREST's central claim is that dense token-level supervision from a privileged self-teacher can be added to verifier-based policy optimization by splitting the teacher's role: verified rewards set the sign of every per-token gradient, while the self-teacher only modulates its magnitude. A turn-segmented, group-relative advantage $A^{\\mathrm{turn}}_{[t]}$ isolates each turn's own outcome, and an entropy-gated teacher factor $\\phi_t$ reweights tokens inside the turn without ever changing the sign of $A^{\\mathrm{turn}}_{[t]}$. The paper reports that this combination reaches 52.0% average accuracy on BFCL V3 with Qwen3-4B and 9.38% session accuracy on WildToolBench with Qwen3-8B, outperforming all RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics.","pith_inferences":["The same magnitude-only principle may extend to other hierarchical generation tasks, such as multi-hop retrieval, collaborative dialogue, or long-horizon planning, whenever a local verifier can score each segment; the paper names these as future directions but does not test them.","A testable variant is an adaptive or scheduled $\\lambda$: the paper fixes $\\lambda=0.3$ and reports that $\\lambda=0.5$ over-concentrates gradient, so decaying teacher influence across training could improve later-step stability.","Because the entropy gate uses token surprisal as a proxy for importance, replacing it with a learned token-importance estimator while keeping the sign-preserving gates might sharpen the teacher's focus; this is an open variant, not a claim of the paper.","The formal sign-preservation guarantee is token-wise rather than aggregate: reweighting changes how token gradients cancel in the sum, so 'verifier-bounded' should be read as a ceiling on each token's scalar direction rather than a guarantee that the final stationary point is verifier-determined."],"forward_implications":["CREST's direction gate guarantees, per token, that the update sign equals the sign of the turn-segmented verified advantage, so a privileged teacher can be used without replacing the verifier as the optimization target.","The largest measured gains appear on the splits where credit dilution is worst, namely BFCL V3 Long Context and WildToolBench session accuracy, indicating that the method targets the failure regime that limits multi-turn agent training.","Ablations show that inter-turn and intra-turn credit are complementary: neither turn segmentation alone nor teacher modulation alone reaches the combined result, and removing both gates drops performance back to GRPO-level accuracy.","With default settings $\\lambda=0.3$, $\\epsilon=0.28$, the teacher's per-token amplification is bounded by 8.4% of the inter-turn per-token gradient norm, making the teacher's influence a small, controlled perturbation.","On-policy distillation baselines plateau below RL baselines in these experiments, and CREST surpasses the distillation ceiling by step 20 on the reported training-accuracy curve, consistent with the claim that its direction gate decouples teacher magnitude from verifier direction."],"supporting_citations":[{"why":"Supplies the GRPO baseline and the group-relative advantage formula that CREST adapts into a per-turn advantage.","marker":"Shao et al. (2024)"},{"why":"Supplies the on-policy self-distillation method, the privileged ground-truth-conditioned teacher construction, and the stabilization recipe CREST inherits.","marker":"Zhao et al. (2026)"},{"why":"Documents gradient-concentration collapse and the teacher-bounded ceiling that motivate CREST's direction gate and entropy gate.","marker":"Brown (2026)"},{"why":"Defines on-policy distillation and its teacher-bounded ceiling, used as a baseline and as the contrast for CREST's magnitude-only teacher.","marker":"Lu & Thinking Machines (2025)"},{"why":"Provides the MT-GRPO baseline, a per-agent-step advantage method that still lacks intra-step token credit assignment.","marker":"Wei et al. (2025b)"},{"why":"Provides the EnvTuning process-reward baseline and is named as a future direction for combining finer reward signals with CREST.","marker":"Lu et al. (2026a)"},{"why":"Contributes the BFCL V3 benchmark and its multi-turn split, one of the two evaluation protocols.","marker":"Patil et al. (2025)"},{"why":"Contributes the WildToolBench benchmark and the observation of low session accuracy that motivates multi-turn credit assignment.","marker":"Yu et al. (2026a)"},{"why":"Supplies the Qwen3 model family used at the two model scales across all training methods.","marker":"Yang et al. (2025)"},{"why":"Supports the premise that self-distillation ceilings are teacher-bounded, which motivates anchoring gradient direction to verified rewards.","marker":"Kim et al. (2026)"}],"fun_headline_variants":["Sign from verifier, size from teacher lifts tool agents","CrEST: dense credit assignment that respects verifier bounds","Teacher modulates, verifier bounds: two-level credit for LLM agents","Beyond trajectory credit: turn-level advantages with teacher scaling","Dense per-token supervision without losing verifier ceiling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that per-turn verifier rewards exist and are semantically aligned across the rollouts of a training group, meaning all rollouts share the same fixed session and ordered user turns so that turn $k$ denotes the same benchmark request; if rewards are terminal-only or turn structures vary, the turn-segmented advantage no longer isolates one turn's credit and the mechanism collapses.","fun_headline_variants_meta":{"raw":{"variants":["Sign from verifier, size from teacher lifts tool agents","CrEST: dense credit assignment that respects verifier bounds","Teacher modulates, verifier bounds: two-level credit for LLM agents","Beyond trajectory credit: turn-level advantages with teacher scaling","Dense per-token supervision without losing verifier ceiling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3933,"prompt_tokens":936,"completion_tokens":2997,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2914}},"tokens_in":552,"tokens_out":2997,"duration_ms":18348,"temperature":1.0,"reasoning_tokens":2914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:58:57.852679+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CREST on a multi-turn benchmark where only session-level success is observable, or where rollouts within a group have different turn counts and orders; if the method then no longer beats GRPO and MT-GRPO on the held-out splits, the turn-alignment assumption fails. A second check is to match the OPSD setup and measure the top-5% token gradient share: if CREST's share approaches OPSD's roughly 77% instead of the reported middle ground, the entropy gate is not preventing concentration collapse.","supporting_citations":[],"review_version":1}