Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Penalize the path, reward the outcome

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 10:54 UTC pith:MQ57VNWI

load-bearing objection Useful conceptual framing with honest experiments; main gap is the near-floor success regime on the real benchmark. the 3 major comments →

arxiv 2607.07435 v1 pith:MQ57VNWI submitted 2026-07-08 cs.LG cs.AI

RLVP: Penalize the Path, Reward the Outcome

Bojie Li , Noah Shi This is my paper
classification cs.LG cs.AI
keywords reinforcement learning from verifiable rewardsgroup-relative policy optimizationprocess rewardreward shapingagent safetywithin-group varianceverifiable penaltysample efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when training agents to act in real, costly, irreversible environments (like placing phone calls or running shell commands), outcome-only reinforcement learning is blind to two problems: it cannot express outcome-neutral deployment constraints (e.g., do not call outside business hours, do not run destructive commands), and it wastes expensive rollouts on all-fail groups where every trajectory receives identical reward and the gradient is zero. The authors observe that real environments are asymmetric verifiers: they can cheaply and reliably detect bad actions but cannot certify progress. This asymmetry means the natural dense signal is a penalty on specific bad moves, not a reward for partial progress. The paper formalizes this through a variance decomposition: in group-relative RL, a trajectory contributes gradient only when its reward differs from the group mean. On all-fail or all-success groups, the outcome reward has zero variance, so any usable gradient must come from an auxiliary process term. A verifiable per-action penalty supplies this variance reliably because bad actions are always sampled and always detectable. A progress-based potential, by contrast, supplies variance only when the policy actually reaches differing intermediate states, which is often not the case on hard tasks. The paper presents a recipe combining the outcome reward with a verifiable penalty, governed by four design rules: penalize concrete verifiable actions (never the absence of progress), keep the outcome reward as the primary driver, pair each penalty with a fulfillment credit for the compliant action, and seed the compliant path so it is reachable early in training. Without these rules, a penalty in isolation collapses into an inaction trap where the policy stops acting entirely.

Core claim

The central mechanism is that group-relative advantage equals within-group variance, and a binary outcome reward drives this variance to zero on both all-fail groups (early training) and all-success groups (late training). The variance decomposition Var_G(R) = Var_G(O) + beta^2 Var_G(Phi) + 2*beta*Cov_G(O, Phi) shows that when Var_G(O) = 0, all usable gradient must come from the process term's own within-group variance. A verifiable penalty on bad actions satisfies this condition by construction because some rollouts take the bad action and others do not, creating variance even at zero percent task success. A progress potential satisfies it only conditionally, when the policy reaches states.

What carries the argument

Verifiable path channel: a deterministic per-action rule engine that evaluates each action against a predicate over the pre-action state and the action taken, issuing a penalty for verified bad moves and a fulfillment credit for verified compliant moves, combined with the standard outcome reward and normalized separately.

Load-bearing premise

The paper's strongest real-task harm reduction result (a sixfold reduction in harmful actions) is obtained with a small model on a benchmark where task success is near the floor (about 10 percent). The mechanism predicts that penalty variance remains available regardless of outcome level, but this is unvalidated at higher success rates where a more capable policy might stop sampling bad actions frequently enough to maintain the within-group variance the penalty needs.

What would settle it

If a sufficiently capable policy, once it starts succeeding reliably, stops sampling the bad actions the penalty targets often enough to maintain within-group variance, the penalty channel would lose its gradient supply and the mechanism would degrade at exactly the capability level where deployment matters most.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents deployed in irreversible real-world settings (phone calls, financial transactions) could learn to respect operational constraints like business hours and authentication requirements that outcome rewards structurally cannot express, reducing harmful behavior without sacrificing task success.
  • The reachability diagnostic (measuring within-group variance of a process signal on base-policy rollouts before training) gives practitioners a cheap pre-training test for whether a dense potential will help or be vacuous, avoiding wasted compute on unreachable signals.
  • The finding that learned critics are inert as training rewards even with near-perfect recall (due to small false-positive rates) suggests that the field's investment in learned reward models for agentic RL may be better redirected toward verifiable rule-based penalties.
  • The inaction trap result implies that any safety-motivated penalty mechanism deployed without a competing outcome reward will reliably produce a passive, useless policy, clarifying the design space for safe RL.
  • If the variance-based mechanism holds at higher task success rates (currently validated only near the floor), it would mean that penalty-based constraint enforcement scales with capability rather than degrading as the policy becomes competent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The variance decomposition is general and model-agnostic, so the same penalty channel should in principle supply gradient on all-success groups (late training, high capability) just as it does on all-fail groups, but this prediction is unvalidated because the paper's strongest real-task result is at near-floor success rates where the policy may behave differently than a competent one.
  • If a capable policy converges to uniformly compliant behavior before the outcome reward differentiates trajectories, the penalty's within-group variance could collapse, potentially making the mechanism self-limiting at high capability rather than universally available as the paper's framing suggests.
  • The four design rules (penalize commission not omission, keep outcome as driver, pair with fulfillment, seed reachability) constitute an implicit claim that the space of effective penalty configurations is narrow, which could be tested by systematically violating each rule and measuring the failure mode, the paper partially does this but a complete factorial study is absent.
  • The verifier-asymmetry principle could extend beyond penalties to any environment where detection is cheaper than certification, such as safety filters in code generation (detecting vulnerabilities is easier than proving correctness), suggesting a broader reallocation of verification effort across AI safety.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript introduces RLVP, a method for group-relative RL that adds a per-action verifiable penalty channel alongside the standard outcome reward. The central insight is a within-group variance decomposition (Eq. 1): on all-fail or all-success groups, the outcome reward provides zero gradient, so any useful dense signal must supply its own within-group variance. The paper argues that verifiable penalties on bad actions reliably supply this variance (since bad actions are frequently sampled early in training), while progress potentials do so only where partial progress is reachable. The method is governed by four design rules (penalize commission not omission, keep outcome as driver, pair penalty with fulfillment credit, ensure reachability). Experiments span synthetic system-administration tasks, TerminalBench (harm reduction at ~10% success), miniF2F theorem proving (sample efficiency with aligned potentials), and SWE-bench/SWE-smith (reachability diagnostics). The code is publicly available.

Significance. The variance decomposition (Eq. 1) is a mathematical identity, not a fitted result, and provides a clean, falsifiable condition for when dense signals help in group-relative RL. The reachability diagnostic—measuring Var_G(Φ) on base-policy rollouts before training—is a practically useful, parameter-free test. The four design rules are derived from ablations (Figure 7), not assumed. The TerminalBench result (Table 1: 0.66 vs 3.71 violations/episode at equal success) demonstrates a real, if narrow, harm-reduction effect. The honest reporting of single-run speed-ups that failed to replicate under re-seeding (§A.2) is commendable. The paper ships reproducible code and a pre-registered un-gameability sweep (Appendix B.1).

major comments (3)
  1. §2, Figure 3, and Abstract: The paper repeatedly claims that a verifiable penalty 'reliably' or 'always' supplies within-group variance because 'bad actions are always detectable.' But detectability ≠ sampling. Var_G(Φ) > 0 requires that some rollouts in a group take the bad action while others do not. As the policy converges toward compliant behavior, Var_G(Φ) → 0, and the penalty becomes as blind as the outcome. The paper implicitly acknowledges this by annealing the penalty after compliance saturates (§3.1, §A.3 Figure 17), which creates a race condition: the penalty must teach compliance before the policy converges to a successful-but-violating behavior from which it no longer samples bad actions. On TerminalBench (Table 1, ~10% success), the policy never approaches convergence, making this the easiest possible regime for the penalty. The abstract's framing of the penalty as 'always'
  2. §3.2, Table 1: The strongest real-task result is at near-floor success (4B on TerminalBench, 0.097 vs 0.122, equal within 1σ). The paper acknowledges this in Limitations. However, the variance decomposition predicts that penalty variance 'remains reachable regardless of outcome level'—a prediction that is unvalidated at higher success rates where a capable policy may converge to uniformly compliant behavior before the outcome reward differentiates trajectories. This is load-bearing for the central claim that the penalty is 'always reachable,' yet the only high-success-rate evidence is on synthetic tasks (Figure 5: 0.96–0.99 success), where the penalty and outcome work together rather than competing. The untested regime is a real benchmark at 30–50% success. The paper should either temper the 'always reachable' framing to match what was validated or provide evidence at above-floor success
  3. §4, Figure 10, Table 2: The 30B results conflate two variables—the aligned potential and the Muon optimizer. The paper states that 'a bounded optimizer (Muon) is necessary to convert the extra gradient from a dense potential into stable success' (§4, §A.3), making this an ad-hoc axiom rather than a derived consequence. The 30B outcome-only configuration diverges on 3/5 seeds under Muon but is stable under AdamW at 19.2 iterations. The aligned potential is stable under Muon at 5.4 iterations. But without an aligned-potential + AdamW cell, it is unclear whether the potential itself provides stability or whether Muon is doing the work. This is a gap in the experimental matrix that weakens the claim that the potential, rather than the optimizer choice, is the decisive factor
minor comments (5)
  1. §7, first line: 'TThe' typo in the conclusion's opening sentence.
  2. Table 1: The violations/episode standard deviations are large relative to the means (0.66±0.63 for RLVP), suggesting high seed variability. A per-seed breakdown or a note on the distribution shape would help readers assess robustness.
  3. Figure 3: The claim that the penalty 'reliably provides this signal' is illustrated conceptually but not backed by empirical data on Var_G(Φ) over training. A plot showing Var_G(Φ) for the penalty channel across training iterations (analogous to the reachability diagnostic in Figure 13) would directly support the central claim.
  4. §A.2: The reachability map (Figure 13a) notes that 'we could not fill the map's interior with controlled training points.' This limits the strength of the benefit-vs-reachability correlation in Figure 13b to a few anchors. This should be acknowledged more prominently in the main text, not only in the appendix.
  5. The paper uses 'reliably' and 'always' in multiple load-bearing contexts (abstract, §2, §3) where the evidence supports 'in the regimes tested.' Aligning the language with the evidence would strengthen the contribution.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and substantive reading. The three major comments each identify genuine issues: (1) the abstract and §2 overclaim by framing the penalty as 'always' supplying variance, when in fact variance decays as the policy converges toward compliance; (2) the strongest real-task result is at near-floor success, and the 'always reachable' prediction is unvalidated at moderate success rates on real benchmarks; and (3) the 30B experimental matrix is missing the aligned-potential + AdamW cell, leaving it unclear whether the potential or the optimizer is responsible for stability. We agree with all three and will revise accordingly: tempering the 'always' framing to match what was validated, adding an explicit discussion of the convergence/decay regime and the race condition it creates, and running the missing optimizer × potential cell at 30B. The variance decomposition itself (Eq. 1) is a mathematical identity and is not in dispute; the disagreement is about how far the empirical evidence supports the strong framing in the abstract.

read point-by-point responses

  1. Referee: §2, Figure 3, and Abstract: The paper repeatedly claims that a verifiable penalty 'reliably' or 'always' supplies within-group variance because 'bad actions are always detectable.' But detectability ≠ sampling. Var_G(Φ) > 0 requires that some rollouts in a group take the bad action while others do not. As the policy converges toward compliant behavior, Var_G(Φ) → 0, and the penalty becomes as blind as the outcome. The paper implicitly acknowledges this by annealing the penalty after compliance saturates (§3.1, §A.3 Figure 17), which creates a race condition: the penalty must teach compliance before the policy converges to a successful-but-violating behavior from which it no longer samples bad actions. On TerminalBench (Table 1, ~10% success), the policy never approaches convergence, making this the easiest possible regime for the penalty. The abstract's framing of the penalty as 'always'

    Authors: The referee is correct on all counts. Detectability is necessary but not sufficient for within-group variance; what is required is that the policy still samples both the violating and non-violating action within the same group. As compliance saturates, Var_G(Φ) → 0 and the penalty channel goes blind, exactly as the outcome does on all-succeed groups. This is not a corner case—it is the expected asymptotic behavior, and our abstract and §2 overclaim by eliding it. The annealing schedule (§3.1, §A.3) is indeed an implicit acknowledgment of this decay, and the referee is right to flag the resulting race condition: the penalty must teach compliance before the policy converges to a successful-but-violating fixed point from which bad actions are no longer sampled. We do not currently have evidence that this race is always won—it is won in our experiments (where compliance saturates before success, and the annealing removes the penalty before it can interfere), but this is a property of the specific task difficulty and model capability, not a universal guarantee. We will revise the abstract, §2, and Figure 3 to replace 'always' and 'reliably' with the precise condition: the penalty supplies within-group variance as long as the policy samples a mix of violating and compliant actions, which holds during the learning phase but decays as compliance saturates. We will add an explicit paragraph in §3.1 discussing the convergence regime, the race condition, and the role of annealing as the mechanism that manages it. The TerminalBench result being the 'easiest possible regime' is also correct and we will say so explicitly. revision: yes

  2. Referee: §3.2, Table 1: The strongest real-task result is at near-floor success (4B on TerminalBench, 0.097 vs 0.122, equal within 1σ). The paper acknowledges this in Limitations. However, the variance decomposition predicts that penalty variance 'remains reachable regardless of outcome level'—a prediction that is unvalidated at higher success rates where a capable policy may converge to uniformly compliant behavior before the outcome reward differentiates trajectories. This is load-bearing for the central claim that the penalty is 'always reachable,' yet the only high-success-rate evidence is on synthetic tasks (Figure 5: 0.96–0.99 success), where the penalty and outcome work together rather than competing. The untested regime is a real benchmark at 30–50% success. The paper should either temper the 'always reachable' framing to match what was validated or provide evidence at above-floor success

    Authors: This is correct and we cannot fully answer it with the current evidence. The prediction that penalty variance 'remains reachable regardless of outcome level' is supported by the synthetic tasks (Figure 5, 0.96–0.99 success) but, as the referee notes, in those tasks the penalty and outcome are aligned rather than competing—the policy reaches high success through compliant behavior, so the penalty does not face the adversarial regime where a capable policy converges to successful-but-violating behavior. The real-benchmark evidence (TerminalBench, Table 1) is at near-floor success, which is the easiest regime for the penalty. The intermediate regime—a real benchmark at 30–50% success where the policy is capable enough to succeed but might converge to a violating strategy—is genuinely untested. We do not have this experiment and cannot honestly claim the result. We will take the first option the referee offers: temper the 'always reachable' framing throughout the paper (abstract, §2, §3, §5, Limitations) to match what was validated. Specifically, we will state that the penalty supplies variance during the learning phase and in the low-success regime (validated on TerminalBench), that it continues to supply variance at high success when compliance and success are aligned (validated on synthetic tasks), but that the adversarial intermediate regime—where a capable policy could converge to successful-but-violating behavior before the penalty teaches compliance—remains unvalidated on real benchmarks. We will add this as an explicit open question in Limitations rather than leaving the 'always reachable' claim standing. revision: yes

  3. Referee: §4, Figure 10, Table 2: The 30B results conflate two variables—the aligned potential and the Muon optimizer. The paper states that 'a bounded optimizer (Muon) is necessary to convert the extra gradient from a dense potential into stable success' (§4, §A.3), making this an ad-hoc axiom rather than a derived consequence. The 30B outcome-only configuration diverges on 3/5 seeds under Muon but is stable under AdamW at 19.2 iterations. The aligned potential is stable under Muon at 5.4 iterations. But without an aligned-potential + AdamW cell, it is unclear whether the potential itself provides stability or whether Muon is doing the work. This is a gap in the experimental matrix that weakens the claim that the potential, rather than the optimizer choice, is the decisive factor

    Authors: The referee is correct that the 30B matrix is missing the aligned-potential + AdamW cell, and that without it we cannot distinguish whether the potential itself provides stability or whether Muon is doing the work. The current matrix has three of the four cells (aligned potential + Muon: stable and fast; outcome-only + Muon: diverges on 3/5; outcome-only + AdamW: stable but slow) but the fourth (aligned potential + AdamW) is absent. This is a genuine gap. We will run this cell at 30B with 5 seeds and add it to Table 2 and Figure 10. We can commit to running this experiment; however, we cannot report the result in this response because it has not yet been run. We will also soften the claim in §4 and §A.3 from 'a bounded optimizer is necessary' (which is stronger than what the current matrix supports) to a statement of what is actually shown: under Muon, the aligned potential is stable while outcome-only is not; under AdamW, outcome-only is stable but slow. Whether the potential provides stability independent of the optimizer is precisely the question the missing cell addresses, and we will frame it as such rather than as a settled axiom. If the aligned-potential + AdamW cell turns out stable and fast, the claim shifts to 'the potential provides stability, and Muon provides speed.' If it is stable but slow, the claim becomes 'the potential provides stability, and Muon provides speed.' If it is unstable, the referee's concern is validated and we will say so. We will report whichever result obtains. revision: yes

Circularity Check

0 steps flagged

No significant circularity; variance decomposition is a mathematical identity, design rules come from ablations, reachability diagnostic is pre-registered and validated

full rationale

The paper's central derivation chain is self-contained and not circular. (1) Equation 1, Var_G(R) = Var_G(O) + β²Var_G(Φ) + 2βCov_G(O,Φ), is a standard algebraic identity for the variance of a sum R = O + βΦ — not a fitted result or a definition that presupposes its conclusion. The claim that on all-fail/all-success groups all usable gradient must come from Var_G(Φ) follows directly from Var_G(O) = 0, which is a property of binary outcomes, not an assumption smuggled in. (2) The penalty channel is a deterministic predicate over state and action (§3.1: 'a pure predicate over the pre-action state and the action taken'), not a learned model fitted to the data it then predicts. (3) The four design rules are derived from ablations (Figure 7, Figure 20), not assumed as premises. (4) The reachability diagnostic — measuring Var_G(Φ) on base-policy rollouts before training — is computed independently of training outcomes and then validated against them (Figure 13), which is a genuine falsifiable prediction rather than a fit renamed as a result. (5) Self-citations are minimal: Pine AI [2026a, 2026b] are institutional/workflow references, not load-bearing for any derivation. No prior-work uniqueness theorem or ansatz is invoked to force the present paper's choices. (6) The paper is notably self-aware about the one place where a result could appear tautological — Figure 19, where the fulfillment credit rewards test-running and productive edits. The paper explicitly states: 'the fulfillment credit rewards test-running and productive edits, so part of their rise is by construction; the non-circular evidence is that the penalized behaviors (which the fulfillment credit does not touch) also fall, and that the outcome baseline is perfectly flat.' This is the paper identifying and addressing a potential circularity, not committing one. The skeptic's concern about Var_G(Φ) → 0 as the policy converges is a validity/generalization risk (untested at above-floor success rates), not a circularity in the derivation. Score 1 reflects the absence of circular steps with a minor note that the fulfillment-credit result in Figure 19 has a partially by-construction component that the paper transparently flags.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). It introduces a method (verifiable path channel) and design rules, but these are techniques, not postulated objects. The free parameters (λ, β, normalizers, annealing schedule) are standard RL hyperparameters, not constants fitted to reproduce a target result. The axioms are mostly standard math or domain assumptions; the Muon requirement is the most ad hoc element.

free parameters (4)
  • λ (penalty magnitude) = not specified numerically
    Per-action penalty for verified bad moves. The paper does not give a specific value; it is treated as a design choice governed by the four rules. Not fitted to data in the regression sense, but chosen per domain.
  • β (fulfillment/progress credit magnitude) = not specified numerically
    Credit for verified good moves or progress. Same status as λ: a hand-set hyperparameter, not a fitted constant.
  • Per-channel normalizers = not specified
    The two channels (outcome and path) are normalized separately. The normalization scheme is described but not parameterized.
  • Annealing schedule = not specified
    Path shaping is annealed once compliance saturates. The schedule is mentioned but not specified; the ablation (Table 2) shows annealing is unnecessary for the potential.
axioms (4)
  • standard math Group-relative advantage is equivalent to within-group variance.
    §2: This is a mathematical identity for GRPO-style methods where the baseline is the group mean. It is correct and not specific to this paper.
  • domain assumption Real agentic environments are asymmetric verifiers: they can cheaply detect bad moves but cannot certify progress.
    §1, §5: This is the central domain assumption. It is plausible for the settings described (phone calls, shell commands, theorem proving) but is an empirical claim about the structure of deployment environments, not a proven fact.
  • domain assumption The transfer from proxy benchmarks to live deployment is mechanistic, not numerical.
    §5 (Discussion): The paper claims that the same principles, not the same numerical results, will apply in live production traffic. This is unvalidated.
  • ad hoc to paper A bounded optimizer (Muon) is necessary to convert the extra gradient from a dense potential into stable success.
    §4, §A.3: The paper finds that aggressive optimizers destabilize the densely shaped policy. This is presented as a practical finding rather than a derived result.

pith-pipeline@v1.1.0-glm · 26817 in / 3056 out tokens · 606282 ms · 2026-07-09T10:54:08.622153+00:00 · methodology

0 comments
read the original abstract

Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.

Figures

Figures reproduced from arXiv: 2607.07435 by Bojie Li, Noah Shi.

Figure 1
Figure 1. Figure 1: Two verifiable path signals for agents that learn from costly, real-world rollouts. (a) Penalize the path: rewarding the outcome while penalizing the path drives violation-free episodes from near zero to ≈100%—a deployability constraint outcome-only GRPO is blind to and never reaches—at no cost to task success (§3). (b) Reward progress where reachable: a dense potential reaches competence in fewer costly r… view at source ↗
Figure 2
Figure 2. Figure 2: Why the path channel creates gradient where the outcome cannot. Four rollouts of the same bug-fix task, all of which fail the hidden test (outcome is 0 for every trajectory). Left (outcome-only): All four rollouts receive identical reward equal to the group mean. The advantage is zero everywhere, so the policy update is dead. Right (with the verifiable path channel): The same rollouts receive different pat… view at source ↗
Figure 3
Figure 3. Figure 3: Outcome-only RL is blind at both ends of the success rate. Group-relative advantage equals within-group variance. A binary outcome signal drives this variance to zero both on all-fail groups (early in training) and all-success groups (late in training). A dense process signal is useful only insofar as it supplies variance in these regimes—and only in states the policy can actually reach. A verifiable penal… view at source ↗
Figure 4
Figure 4. Figure 4: The verifiable path channel in the real-world (phone-agent) setting. Left (RLVR): Outcome￾only training receives only a terminal reward and remains blind to the path. Violations such as over-calling or acting without proper authentication go unpenalized. Right (RLVP): A deterministic per-action rule engine attaches an immediate verifiable signal to the responsible action—a penalty −λ for bad moves (e.g., o… view at source ↗
Figure 5
Figure 5. Figure 5: Rewarding the outcome while penalizing the path satisfies both task success and constraints. Outcome-only training solves the task but violates outcome-neutral rules on nearly every episode. The two-channel method maintains high success while achieving near-100% violation-free episodes across tasks and model scales (mean ± std over 5 seeds) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A design procedure for verifiable penalties. Before introducing a penalty, each rule can be checked against four criteria: (1) Is there a concrete, verifiable action to penalize (rather than the absence of progress)? (2) Does the outcome reward remain the primary driver of task completion? (3) Is the penalty paired with a corresponding credit for the compliant action? (4) Is the compliant action reachable … view at source ↗
Figure 7
Figure 7. Figure 7: Each design element earns its place. On diverse held-out tasks, outcome-only training solves the task but exhibits high violation rates (low clean rate). Adding the verifiable penalty channel without its complementary fulfillment credit and reachability seeding fails to reliably produce both clean and successful behavior across seeds. Only the full recipe—penalty + fulfillment + seeded compliant demonstrat… view at source ↗
Figure 8
Figure 8. Figure 8: The aligned potential in action on a real MINIF2F theorem. On mathd_algebra_109, the Lean kernel verifies each tactic and the proof’s remaining obligations fall 3→2→1→0. The outcome reward is sparse: it pays a single +1 only when the proof is completed (■). The aligned potential is dense: every kernel-verified drop in obligations pays a +β fulfillment credit (while an errored tactic in another rollout pays… view at source ↗
Figure 9
Figure 9. Figure 9: Dense potentials are reachability-gated. Left: On software repair, the finer-grained potential is unreachable. Across many rollouts, the policy makes zero partial progress, yielding zero within-group variance and thus no gradient. Right: Where the potential is reachable, it eliminates the dead all-fail updates that dominate early training (0 vs. ∼16 of 40 iterations per seed). It supplies gradient precisel… view at source ↗
Figure 10
Figure 10. Figure 10: The aligned-potential recipe on real theorems (miniF2F algebra), five seeds per config￾uration. Thin lines show individual seeds; thick lines show the mean over stable seeds; × marks a divergence-guard abort. (a) Controlled same-optimizer comparison at 4B scale: the aligned potential crosses the 0.9 threshold in 4.4 ± 0.5 iterations vs. 7.0 ± 0.7 for outcome-only. It is faster on every seed and never dive… view at source ↗
Figure 11
Figure 11. Figure 11: Verifiable agentic RL is often verifier-bound. In theorem-proving experiments the GPU sits near-idle at full memory utilization while the CPU-based proof kernel verifies each step. The same signal that makes a penalty cheap also moves the bottleneck away from the accelerator. A systems note. The same property that makes verifiable penalties cheap—a machine￾checkable environment—often makes the verifier a … view at source ↗
Figure 12
Figure 12. Figure 12: In software repair the finer potential is structurally rare and empirically unreachable. (a) Of 48 single-file-fix SWE-bench instances, two-thirds have a single failing test, so the test-fraction potential equals the binary outcome; only one-third admit a finer Φ. (b) For those, a 30B policy’s rollouts never realize it: all 156 rollouts score Φ = 0. The band of intermediate values exists but is empty—zero… view at source ↗
Figure 13
Figure 13. Figure 13: Reachability, measured cheaply, gates the dense-reward benefit. (a) VarG(Φ) on base-policy rollouts (no training) over model capability × outcome sparsity: it rises with capability and peaks in the sparse-but-reachable corner. (b) The benefit of a dense reward (final-success gap, dense − outcome) against that same probe: zero where VarG(Φ)≈0 and large where it is high. Benefit appears only past a reachabi… view at source ↗
Figure 14
Figure 14. Figure 14: Dead updates across two regimes. Fraction of iterations with an all-fail, zero-spread group (Qwen3-4B, 3-seed mean). Chained task (bottleneck reachable): outcome-only 65%, DAPO 54% at a 5.6× generation tax, the aligned potential 8%. Silent precondition gate (§A.4, pivotal action never sampled): outcome and DAPO dead on every iteration, the reward-only potential 76%; only a demonstration seed breaks the wa… view at source ↗
Figure 15
Figure 15. Figure 15: Sample efficiency on the calibrated hard task (N=4). Held-out success vs. episodes gener￾ated. Outcome-only sits near base rate for most of training; the aligned potential climbs immediately from the all-fail groups; dense baselines also outpace outcome-only. The x-axis counts episodes gener￾ated, including DAPO’s resampling [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Efficiency over three seeds. Left: episodes to the success threshold (lower better)—the potential is several times faster than GRPO and DAPO. Right: the mechanism—GRPO and DAPO burn most iterations on dead or stalled updates, and DAPO generates 5.6× the episodes it uses. Note the large GRPO error bar: the all-fail lottery on the outcome side, which the dense potential’s reachable within-group variance avo… view at source ↗
Figure 17
Figure 17. Figure 17: Component attribution for the potential on N=4 (episodes to threshold, converged success annotated). The token-attached credit channel is the efficiency lever (folding into a scalar return doubles the cost); annealing protects the ceiling; the clean recipe is best. A per-step length cost, useful only on very long horizons, hurts here. appears to regularize where the bursty sparse signal blows up under the… view at source ↗
Figure 18
Figure 18. Figure 18: A non-saturating ceiling test, and the discovery wall. Left: on the silent-gate task every reward-only method (outcome, DAPO, clean potential, even rules-in-prompt) converges to zero— the precondition is never sampled, so the fulfillment credit never fires. Right: a single synthesized demonstration through the process channel breaks the wall to perfect held-out success, a sharp phase transition. clean pot… view at source ↗
Figure 19
Figure 19. Figure 19: The process reward learns where the outcome is dead (SWE-smith, 8B, ≈0% solve). (a) Productive actions per episode over training, two seeds: outcome-only is flat (the all-fail gradient is dead), while the process reward rises steadily. (b) Final-3 trajectory quality (mean of two seeds): productive edits and test-runs increase several-fold under the process reward, while penalized behaviors (repeat-errors,… view at source ↗
Figure 20
Figure 20. Figure 20: Which signals survive (Qwen3-30B, miniF2F, 3 seeds). (a) Terminal success (last-3-iteration mean) for five signals, mean ± std with seed dots. The three penalty-free signals sit well above the dead zone on every seed; the pure penalty is reliably dead; the fulfillment+penalty configuration has by far the largest error bar. (b) That configuration is bimodal—collapsing on two seeds, rescued to 1.0 on one—wh… view at source ↗
Figure 21
Figure 21. Figure 21: The coverage gradient on τ-bench airline. Left: training reward for four signals. Outcome￾only learns the benchmark; generic rules orthogonal to the reward collapse into the inaction trap; policy-derived procedural and verifiable-semantic rules (with early annealing) do not. Right: mean reward per tier (bars) with peak (ticks). Misaligned rules harm; aligned rules remove the harm and nearly reach outcome-… view at source ↗
Figure 22
Figure 22. Figure 22: The boundary on one axis: are the rules outcome-instrumental? When the verifiable rules are aligned with what success requires (chained and gated tasks: the workflow is the task), the process channel converts compliance into large outcome gains. When orthogonal to the reward (τ-bench, generic rules), the same machinery optimizes compliance into a degenerate policy. (fetch the user id and look up the reser… view at source ↗
Figure 23
Figure 23. Figure 23: When a verifiable rule beats a same-model self-critic, and when neither trains. Axes: whether a verifiable rule is cheaply specifiable, and whether the on-policy critic can detect the violation. The learned critic is a poor training reward in every quadrant; its only robust use is offline diagnosis at the intent frontier. B.3 A learned critic relocates the hack If verifiable rules cannot reach intent, why… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 39 canonical work pages · 24 internal anchors

  1. [1]

    VeriGate: Verifier-Gated Step-Level Supervision for GRPO

    Aakriti Agrawal, Minghui Liu, and Furong Huang. VeriGate: Verifier-gated step-level supervi- sion for GRPO.arXiv preprint arXiv:2605.30451,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2-bench: Evalu- ating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Stop summation: Min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275,

    Jie Cheng et al. Stop summation: Min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275,

    work page arXiv

  5. [5]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Group-in-Group Policy Optimization for LLM Agent Training

    Lang Feng et al. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.arXiv preprint arXiv:2504.07086,

    Andreas Hochlehnert et al. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.arXiv preprint arXiv:2504.07086,

    work page arXiv

  8. [8]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    https://mp.weixin.qq.com/s/ tJAinVKzZAqZrhiEvGU2Dg(accessed 2026-06-25). Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR),

    work page 2026

  9. [9]

    VinePPO: Refining Credit Assignment in RL Training of LLMs

    Amirhossein Kazemnejad et al. VinePPO: Unlocking RL potential for LLM reasoning through refined credit assignment.arXiv preprint arXiv:2410.01679,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. Tülu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang

    Thanh-Long V . Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in LLM reinforcement learning via entropy- guided advantage shaping.arXiv preprint arXiv:2509.21880,

    work page arXiv

  12. [12]

    RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

    RL VP: Penalize the Path, Reward the Outcome 17 Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning

    Zhiwei Li, Yong Hu, and Wenqing Wang. Encouraging good processes without the need for good answers: Reinforcement learning for LLM agent planning.arXiv preprint arXiv:2508.19598,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Agentic reinforcement learning with implicit step rewards.arXiv preprint arXiv:2509.19199, 2025a

    Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewards.arXiv preprint arXiv:2509.19199, 2025a. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprin...

    work page arXiv

  15. [15]

    NGRPO: Negative-enhanced group relative policy optimization.arXiv preprint arXiv:2509.18851,

    Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Yadong Li, Xingzhong Xu, et al. NGRPO: Negative-enhanced group relative policy optimization.arXiv preprint arXiv:2509.18851,

    work page arXiv

  16. [16]

    Pine AI. Pine AI. https://www.19pine.ai, 2026a. An AI assistant that completes real-world tasks—phone calls, customer service, account and subscription management—on a user’s behalf. Accessed 2026-07. Pine AI. Pine AI: The most natural human-computer interface is your voice. Blog post, 2026b. URL https://www.19pine.ai/blog/ pine-ai-the-most-natural-human-...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [17]

    Spurious Rewards: Rethinking Training Signals in RLVR

    Rulin Shao et al. Spurious rewards: Rethinking training signals in RLVR.arXiv preprint arXiv:2506.10947,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Hybrid reinforcement: When reward is sparse, it’s better to be dense.arXiv preprint arXiv:2510.07242,

    Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Yixuan Li, Jason Weston, and Ping Yu. Hybrid reinforcement: When reward is sparse, it’s better to be dense.arXiv preprint arXiv:2510.07242,

    work page arXiv

  20. [20]

    Qwen3 Technical Report

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

    Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. SPA-RL: Reinforcing LLM agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732, 2025a. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. I...

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    A practitioner’s guide to multi-turn agentic rein- forcement learning.arXiv preprint arXiv:2510.01132,

    Ruiyi Wang and Prithviraj Ammanabrolu. A practitioner’s guide to multi-turn agentic rein- forcement learning.arXiv preprint arXiv:2510.01132,

    work page arXiv

  23. [23]

    Reinforcement Learning for Reasoning in Large Language Models with One Training Example

    Yiping Wang et al. Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571, 2025b. Quan Wei, Siliang Zeng, Chenliang Li, William Brown, and Mingyi Hong. Reinforcing multi- turn reasoning in LLM agents via turn-level reward design.arXiv preprint arXiv:2505.11821, 2025a. Yuxiang Wei, Olivier Du...

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Learning to Reason under Off-Policy Guidance

    Jianhao Yan et al. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    SWE-smith: Scaling Data for Software Engineering Agents

    John Yang, Kilian Lieret Zhang, et al. SWE-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, et al. Dapo: An open-source llm reinforce- ment learning system at scale.arXiv preprint arXiv:2503.14476,

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning

    RL VP: Penalize the Path, Reward the Outcome 19 Yuanqing Yu et al. Steptool: Enhancing multi-step tool usage in llms through step-grained reinforcement learning.arXiv preprint arXiv:2410.07745,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Free Process Rewards without Process Labels

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels.arXiv preprint arXiv:2412.01981, 2024a. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.arXiv preprint arXiv:2401.10020, 2...

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

    Chenchen Zhang. From reasoning to agentic: Credit assignment in reinforcement learning for large language models.arXiv preprint arXiv:2604.09459,

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685,

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    The surprising effectiveness of negative reinforcement in llm reasoning.arXiv preprint arXiv:2506.01347, 2025

    Xinyu Zhu et al. The surprising effectiveness of negative reinforcement in LLM reasoning. arXiv preprint arXiv:2506.01347,

    work page arXiv

  34. [34]

    This appendix gives the full experiments

    A Full Experiments on Dense Potentials The main text treats the dense progresspotentialas the reachability-gated half of the process signal (§4). This appendix gives the full experiments. The picture is consistent with the variance account: where a potential’s intermediate values arereachableit supplies gradient the outcome lacks and accelerates training,...

    work page 2025

  35. [35]

    On SWE-bench software repair [Jimenez et al., 2024] the natural potential is the fraction of hidden FAIL_TO_PASS tests an episode makes pass

    This can be checkedbeforespending training compute, from a handful of base-policy rollouts. On SWE-bench software repair [Jimenez et al., 2024] the natural potential is the fraction of hidden FAIL_TO_PASS tests an episode makes pass. Two facts make it vacuous (Figure 12). Structurally the finer potential barely exists: two-thirds of instances have a singl...

    work page 2024

  36. [36]

    Benefit appears only past a reachability threshold

    0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 base-rollout VarG( ) Figure 13.Reachability, measured cheaply, gates the dense-reward benefit.(a) VarG(Φ) on base-policy rollouts (no training) over model capability × outcome sparsity: it rises with capability and peaks in the sparse-but-reachable corner.(b)The benefit of a dense reward (final-success...

    work page 2025

  37. [37]

    fragility,

    outcome-only crawls, spending its budget on dead updates, then converges only at the end; the aligned potential rises immediately from the early all-fail groups. Reimplemented dense baselines (GiGPO- style step advantages [Feng et al., 2025], StepTool-style per-call rewards [Yu et al., 2024]) also outpace outcome-only—anyadmissibledense signal helps. The ...

    work page 2025

  38. [38]

    any- valid-tactic

    attributes the speed to the token-attached channel: folding the same rule rewards into the scalar return doubles the episodes to threshold. Annealing the shaping off after saturation protects the final ceiling (unrelaxed pressure drifts the policy toward over-compliant, bloated episodes—a mild inaction-trap tendency). The fulfillment credit is the positiv...

    work page 2026

  39. [39]

    The learned critic is a poor training reward in every quadrant; its only robust use is offline diagnosis at the intent frontier

    detects offline (F1 0.63 vs 0.23) but collapses as a reward (0.0) DIAGNOSE ONLY D hidden / masked failures neither channel has signal; needs outcome or a demo open On-policy self-critic detects the violation? DETECTS from trajectory BLIND Verifiable rule cheaply specifiable? YESNO Figure 23.When a verifiable rule beats a same-model self-critic, and when n...

    work page 2023