The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works

Yu Wang

The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rewarding LLM agents for predicting their next observation backfires catastrophically under GRPO: every trained agent collapses into a degenerate 'dark room' state — and the culprit is the standard-deviation term in advantage normalization.

desk verdict Genuine and practically important failure mode, with a clean but conditional theory; the main causal claim needs an extra control before it's fully pinned. read the letter →

arxiv 2607.21273 v2 pith:7C53S2RC submitted 2026-07-23 cs.LG

Yu Wang This is my paper

classification cs.LG

keywords GRPOLLMagentsrewardshapingpredictiondarkroomproblemgroupnormalizationhackingauxiliaryloss

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that a seemingly safe dense reward — a small per-step bonus for correctly predicting the next observation — destroys policies trained with GRPO, a group-normalized RL algorithm built for LLM agents. Across three model scales on ALFWorld, every run ends in a degenerate absorbing state: prediction accuracy reaches 1.0, task success drops to 0, and episode length pins at the horizon. The paper traces the cause to one line of GRPO's update: dividing each group's returns by their standard deviation. Because success is sparse, many training groups contain only failures; inside such groups, the tiny shaping reward is the only variation, and z-scoring amplifies it into full-scale pressure that pushes the policy into a predictable corner. The single-factor ablation is stark: switching to mean-only normalization (same signal, same everything else) restores success from 0% to baseline parity. The paper then shows the same prediction signal is neutral or harmful as a reward but gains about 20 points as an auxiliary loss, and proposes a variance-profile criterion that predicts which dense signals are safe under group normalization.

What carries the argument

The argument is carried by GRPO's group advantage estimator, which normalizes each group's returns by subtracting the mean and dividing by the standard deviation. Proposition 1 (λ-invariance) shows that when a group contains only failed trajectories, all task returns are equal and the shaping term is the sole within-group variation; after z-scoring, the normalized advantage is independent of the shaping coefficient λ, so reducing the reward's size does not reduce the gradient pressure. Proposition 2 (variance-profile criterion) states that the amplifier only stops when the signal's within-group variance vanishes at saturation — meaning the safety of a dense reward is governed by its variance

What would settle it

During the first 30 training steps, record per-trajectory returns for groups where every episode failed and compute the within-group standard deviation; decompose it into the λ-scaled shaping component versus all other sources (penalties, length, noise). If the non-shaping component is not negligible, the invariance premise is empirically violated. Alternatively, run the identical GRPO setup with group size 16 or 32 so that all-fail groups become rare; if the collapse still occurs, the all-fail-group mechanism is not sufficient.

Watch

Extended reading notes

Core claim

The central discovery is that a potential-based prediction reward, bounded at the return level exactly as the shaping literature promises, is transformed by GRPO's standard-deviation normalization into an unbounded force that drives the policy into a degenerate absorbing 'dark room' state. The mechanism is captured by Proposition 1: in a group where every trajectory fails the task, the z-scored advantage is invariant to the shaping coefficient λ, so the bounded reward becomes full-scale pressure no matter how small λ is. Proposition 2 generalizes this into a variance-profile criterion: what z-scoring amplifies is a dense signal's within-group variance at saturation, so signals whose variance

Load-bearing premise

The collapse is blamed on the premise that within every all-fail training group, the only thing that differs between trajectories is the tiny prediction-shaping term; if other factors (invalid-action penalties, length differences, stochasticity) create comparable within-group variation, the λ-invariance proof and the causal story weaken.

Editorial extensions

If this is right

Boundedness at the return level does not make a dense shaping reward safe under GRPO; the advantage normalizer can amplify a λ-scaled term into full-scale gradients.
Removing GRPO's std normalization (mean-only) is sufficient to rescue the policy from collapse, turning 0% into baseline parity (51.6%) and replicating the pathology-free regime.
For sparse-success LLM agents, the auxiliary-loss channel is a safer path for dense prediction signals: identical signal content gains ~20 points over baseline, while every reward-channel variant is at best neutral.
The variance-profile criterion provides a screening rule for dense rewards under group normalization: prefer signals whose within-group variance decays at mastery; avoid potential-difference signals whose per-step variance persists.
Collapse is irreversible once it occurs (constant pressure and annealing both fail), so detection must be preventive; the joint precursor (entropy decline ∧ prediction saturation ∧ length pinning) warns 15–30 steps before the turn.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The λ-invariance mechanism should generalize beyond prediction rewards: any bounded, policy-dependent shaping term that retains within-group variance in all-fail groups should produce the same collapse under GRPO — a directly testable prediction.
If the preregistered disjoint-vocabulary placebo also gains ~20 points, the apparent value of auxiliary prediction losses may be a regularization/compute effect rather than world-model information transfer, which would reframe many auxiliary-objective successes.
A practical mitigation implied by the analysis but not yet run: increasing group size so that all-fail groups become rare should dilute the amplifier; this would both test the mechanism and potentially serve as a fix.
An offline audit could apply the variance-profile criterion to existing dense rewards by computing within-group variance vs. training progress, flagging signals that keep variance at saturation before training runs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First: this paper is worth reading and worth refereeing. The empirical phenomenon is real and important: a dense, bounded, potential-based prediction reward collapses GRPO-trained agents on ALFWorld at three scales, and deleting the std normalization term flips the same reward from 0% to ~50% success. That is a concrete, actionable result for anyone doing RL fine-tuning of LLM agents.

What's genuinely new: the formalization of λ-invariance in all-fail groups (Proposition 1) is a nice two-line proof that explains why annealing can't help and why the advantage scale jumps from ±0.1 to ±5–7. The variance-profile criterion is a sensible organizing principle, and the paper tests it prospectively with preregistered predictions, which is more than most empirical papers do. The channel-effect matrix, with the shuffled-gold placebo, is a well-designed attribution experiment. The paper is unusually honest about its own weaknesses: single-seed endpoints, last-6 means, descriptive cross-arm comparisons, the leaky placebo, and the collinearity of group size with the pathology are all stated clearly.

The soft spots are real, but mostly in proportion. The biggest one, to me, is that the premise of Proposition 1 is likely violated in the actual experiments. The proof assumes that within an all-fail group, returns differ only via the shaping term. But the config includes an invalid-action penalty of 0.1 and episodes can terminate early, so trajectories in the same group can differ in non-shaping components. That means the z-scored advantage is not strictly λ-invariant, and the collapse could be amplified by any small within-group variance, not specifically the prediction signal. The paper asserts the shaping term is the only within-group variation, but doesn't measure the variance contributions. This is a load-bearing assumption for the causal attribution. The stress-test note is right to flag it. That said, the empirical ablation still shows the prediction signal matters (the std baseline without the signal doesn't collapse), so the core finding stands; it's the precise mechanism that needs tightening.

Minor: the 8B baseline anomaly (32.8% vs. 49.5% at 4B) is unexplained and acknowledged; the endpoints are single-seed, so all numeric comparisons are descriptive. These are disclosed, but they still limit what can be concluded.

Who this is for: anyone training LLM agents with GRPO, and researchers working on reward shaping or intrinsic motivation. It deserves a serious referee, with the main request being a control that varies the invalid-action penalty or measures within-group variance components, plus the promised seed replication. If the replication holds and the mechanism is pinned down, this will be a useful paper. I'd send it to peer review as is, with the expectation of major revision.

Referee Report

4 major / 6 minor

Summary. The paper studies the effect of adding a potential-based next-observation prediction reward (a 'PS signal') to GRPO training of Qwen3-1.7B/4B/8B agents in ALFWorld. It reports that every std-normalized run collapses into a degenerate absorbing state: prediction accuracy -> 1.0, task success -> 0, episode length pinned at the horizon. A single-flag ablation (mean-only vs. std normalization) rescues the PS arm to baseline parity. The authors prove a parameter-free λ-invariance result for all-fail groups under the assumption that all task returns in a group are equal to a constant, and use it to propose a variance-profile criterion that predicts which dense signals are safe under std normalization. A signal-delivery matrix compares reward-channel vs. auxiliary-loss-channel delivery of the same signal, finding the loss channel gains ~20 points, with a shuffled-gold placebo matching the true-gold arm. The paper is explicitly a single-seed preprint with preregistered replications in progress.

Significance. If the reported collapse and rescue replicate, the paper identifies an important and practically relevant failure mode: return-level boundedness guarantees for potential-based shaping do not survive group-relative advantage normalization in sparse-success settings. The λ-invariance proposition (Proposition 1) is a clean parameter-free derivation, and the controlled channel matrix, placebo controls, and preregistration ledger are methodologically strong. The paper is unusually candid about its limitations. However, the current evidence base is single-seed, and a central assumption of Proposition 1 is contradicted by the experiment's own invalid-action penalty and early-termination configuration. These issues prevent the claims from being accepted as stated.

major comments (4)

[§4.3, Eq. (3), and Appendix C] Proposition 1 assumes R_i = C + λs_i for all trajectories in an all-fail group. The experimental configuration includes an invalid-action penalty of 0.1 (Appendix C) and allows early termination. In any all-fail group with different numbers of invalid actions or different lengths, the non-shaping component of return is not constant, so σ_R = sqrt(λ²σ_s² + σ_other²) and the advantage in Eq. (3) is not λ-invariant. The paper asserts (§§4.2, 5) that the shaping term is the 'only within-group variation' in all-fail groups, but no measurement of σ_other is provided; since the penalty magnitude (0.1) equals λ, the std amplifier may be amplifying penalty/termination noise as much as shaping. The no-signal std baseline (49.5%, Table 2) shows the prediction signal is necessary for the collapse, but not that the Proposition-1 mechanism is the operative one. Please add an ablation with no invalid-a
[§4.4 and Proposition 2] The collapse's terminal condition is 'prediction accuracy -> 1.0'. If the rewarded score Φ_t saturates to a constant, the per-step reward r_pred(t)=λ(Φ_t−Φ_{t−1}) becomes zero and the within-group variance σ_s would vanish, which by Proposition 2 should starve the amplifier and prevent the absorbing state. To explain the observed collapse, §4.4 asserts that 'potential-difference scoring keeps fluctuating per step even at prediction accuracy≈1.0', but the manuscript does not define how the logged prediction accuracy relates to the rewarded Φ_t. If Φ_t is a partial-credit F1-like score, this should be stated and the variance trajectory σ_s should be plotted; if Φ_t saturates to a constant, the mechanism is internally inconsistent. This is load-bearing for Proposition 2's retrodictive claims.
[§9 and Table 2] All endpoints are single-seed last-6 validation means, with the text quoting ±8.4 pt single-point noise. Several load-bearing comparisons—the net-attribution ≈0 (§5), the shuffled-gold vs. true-gold loss-channel comparison (76.0 vs. 69.3, §7), and the mean-only vs. decoupled gap (51.6 vs. 31.2, §5)—are descriptively assessed without seed replication. The authors' disclosure is honest and the preregistration is exemplary, but the central empirical claims (three-scale collapse and std-removal rescue) need replication before the title-level conclusion can be accepted. I would ask for at least the collapse/rescue arms under seeds 42/96 to be included in the revision.
[§5 and Fig. 3] The 'advantage-level dose–response' plots three points from three different algorithms (std full-scale, decoupled capped, mean-only) on a common x-axis. The x-axis values are not uniformly measured: the decoupled cap reports 0.066–0.077, while the mean-only arm's raw advantage magnitude is not reported, and the resulting ordering is non-monotonic (mean-only at ~0.1 gives 51.6%, decoupled at ~0.07 gives 31.2%). As presented this is a schematic comparison, not a dose–response curve; please report measured per-arm advantage magnitudes or relabel it accordingly.

minor comments (6)

[Abstract and §1] The phrase 'every run' overstates the evidence; all runs are single-seed. Please qualify as 'every run in our single-seed trials'.
[§9] The 8B baseline anomaly (32.8% vs. 49.5% at 4B) is unexplained. Consider reporting full validation curves for the baselines so readers can judge the scale ordering.
[§3] Please specify whether validation episodes use the same group structure as training and whether the invalid-action penalty applies identically to all arms.
[Table 2] The baseline row's 'std' normalization may confuse readers because std normalization alone does not collapse; a one-sentence explanation in the caption would help.
[§4.7] The three false-alarm counterexamples are useful, but the reader should be told how these counterexamples were selected—exhaustively from all runs, or cherry-picked?
[Appendix A] Please add dates and version identifiers for the SHA256-digest preregistration entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Proposition 1 is a parameter-free consequence of the GRPO estimator, and the empirical claims rest on ablations and preregistered predictions rather than on fitted inputs or self-citation.

full rationale

The paper's core derivation chain is self-contained. Proposition 1 is proven directly from the definition of the GRPO normalized advantage (Eq. 1) and the all-fail-group assumption R_i = C + λ s_i; it is algebra, not a fitted parameter or a restatement of the experimental outcome. Proposition 2 is the same formula in the saturation limit and is used to organize retrodictive and prospective predictions; the paper explicitly labels compatibility checks as 'not an independent test' and reports ledger outcomes, including falsified predictions (e.g., weaker-model-collapses-earlier). The causal localization ('0%→51.6%') is a single-factor ablation (std normalization on/off), with a no-signal mean-only control, which is an interventionist comparison rather than a circular definition. There are no load-bearing self-citations: the only self-identifying item is the Zenodo deposit of an earlier version, cited only as provenance, not as evidence. The concern that the invalid-action penalty and early termination could violate the constant-task-return premise is a robustness/correctness issue about the empirical scope of Proposition 1, not a circularity of the derivation itself. No fitted input is renamed as a prediction; no uniqueness theorem is imported from the authors' prior work; no ansatz is smuggled via citation. Thus the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central claim depends on the z-scoring algebra of Proposition 1 (a valid derivation) plus the empirical premise that all-fail groups dominate and contain no other within-group variation. The free parameters are mostly hyperparameters disclosed by the paper; no hidden fitted parameters are apparent. The invented entity is the variance-profile criterion, which is a repackaging of the algebra rather than a new physical entity; it has no independent evidence yet.

free parameters (4)

shaping coefficient λ = 0.1
λ scales the potential-difference reward. Chosen by hand; the paper notes that annealing to λ=0.068 still collapses and predicts invariance across λ values. It is an input, not fitted, but it is a free hyperparameter that the conclusion depends on (in the λσ >> ε regime).
group size n=4 = 4
Small groups make all-fail groups common. The paper explicitly notes group size is collinear with the std pathology and a group-size ablation is preregistered, so the central claim depends on this choice.
GRPO std floor epsilon = 1e-6
The identity in Proposition 1 holds in the λσ >> ε regime; this floor determines when the invariance breaks. It is a framework default, not fitted, but it is a free constant that the derivation depends on.
validation window (last-6 means) = 6
Endpoint selection is a choice; the paper discloses the noise and that formal endpoints come from a 140-game evaluation. This is a free evaluation parameter.

assumptions (3)

domain assumption In all-fail groups, all task returns are equal to a constant C
Proposition 1 requires this; the paper assumes sparse success makes all-fail groups common and that within an all-fail group the only variation is the shaping term. This is the load-bearing premise and is only approximately true.
domain assumption The GRPO advantage estimator is exactly A_i = (R_i - mean(R))/std(R) + eps with group-level normalization
The derivation applies to the specific GRPO estimator; the paper explicitly scopes all claims to group-relative estimation and does not test critic-based PPO.
standard math The potential-difference reward telescopes exactly as Eq. (2) states
This is a telescoping sum identity, standard math, not controversial.

invented entities (1)

The 'variance-profile criterion' as a safety rule for dense signals
purpose: Organizing principle claiming a dense signal is dangerous under std normalization to the extent that its within-group variance survives mastery.
It is a restatement/reformulation of Proposition 1 that is used to retrodict and to make preregistered predictions; no independent falsifiable handle outside the paper is provided yet (predictions are in progress).

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works." pith.science (2026). https://pith.science/paper/7C53S2RC

@misc{pith2026260721273,
  author       = {Pith},
  title        = {Pith review of: The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7C53S2RC}},
  note         = {Machine review of arXiv:2607.21273}
}

read the original abstract

Dense per-step supervision is the standard remedy for sparse-reward long-horizon LLM agents: reward the policy for predicting its next observation, which looks provably safe under potential-based shaping. Published prediction-reward and auxiliary-loss variants report both successes and instabilities; we supply the controlled account: 74 preregistered arms dissect one fixed prediction signal under GRPO across ALFWorld, WebShop, a synthetic POMDP, and Qwen3-1.7B/4B/8B, varying only the delivery mechanism. (1) Every run sustaining this difference-form reward under untouched std normalization (no filtering, dynamic-sampling, or decoupling mitigations) collapses: eleven runs across scales, coefficients, group sizes, and groupings (the floor-bound synthetic environment stalls instead); ALFWorld runs end in an absorbing state (prediction accuracy -> 1.0, success -> 0): the optimizer builds the "dark room". The algebra is one line: in all-fail groups z-scoring cancels the shaping coefficient; removing only std normalization restores baseline parity. (2) A signal's danger is set by its within-group variance trajectory, plus hackability as a second axis; it retrodicts every reward-channel collapse and survives preregistered prospective tests. (3) The same signal as a teacher-forced auxiliary loss is harmless on ALFWorld at 4B, but the gain is not the signal's: content-free placebos as a class match or beat gold at both matched seeds (s0: 78.8 vs 68.6; s42: 67.9 vs 57.9); the auxiliary update is the regularizer. (4) At 8B the recipe turns bistable: gold full-weight locks two of three seeds; every content-free or reduced-weight arm stays healthy. No ALFWorld or WebShop reward-channel variant measurably beats its matched-normalization baseline and no gold signal measurably outperforms its content-free placebo: the delivery channel, not the content, decides; which channel is safe is regime-dependent.