{"id":"a90f764f-26c1-44b9-9a9c-742419a4eb8a","arxiv_id":"2607.16257","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HPO learns language-agent policies by using the Wasserstein distance between the current policy and a hindsight distribution in an intent embedding space, producing low-variance step-level advantages.","lead":"This paper proposes HPO, a reinforcement-learning method for long-horizon language agents that compares the current policy with a \"hindsight\" distribution of successful behavior in a semantic embedding space, using Wasserstein distance for step-level credit assignment. It reports more stable training and gains of 4-7% over GRPO/PPO on search-based QA and TextCraft.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (14) asserts without proof that the Wasserstein potential is a policy gradient; the bounded-variance theorem does not establish that this biased signal optimizes the return.","rationale":"The reader's weakest assumption was the offline distribution shift. My concern is more fundamental: even the on-policy formulation lacks a proof that the Wasserstein potential is a valid surrogate for the policy gradient. The paper's empirical results and code link are meaningful, and the variance bound is technically correct, but the central theoretical bridge from KL ratio to Wasserstein potential is asserted rather than derived. This does not overturn the conditional verdict, because the empirical method may still work, but it sharpens the required revision: either prove Eq. (14) under explicit assumptions, or reframe HPO as a heuristic shaped-advantage method with a bias analysis. The proposed tabular check would directly test whether the claimed gradient identity holds in a setting where all quantities can be computed exactly.","tokens_in":21240,"tokens_out":6957,"duration_ms":72543,"concrete_test":"Implement a small tabular MDP (e.g., 3 states, 2 actions, binary terminal rewards, γ=1). For several non-optimal policies, exactly compute (i) the true gradient ∇J, (ii) the KL-ratio signal E_{ρπ}[(ρhπ/ρπ)∇logπ], and (iii) the HPO signal E_{ρπ}[-f*∇logπ], where f* is the W1 Kantorovich potential between ρπ and ρhπ, both on-policy and fixed-offline. Check whether (iii) has positive cosine similarity with (i) and whether a small gradient step along (iii) increases J. If not, Eq. (14) is not a valid policy-gradient identity and the theoretical claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim hinges on Eq. (14), which states ∇θJ(θ) = E_{ρπ}[-δW1(ρπ,ρhπ)/δρπ ∇logπ] = E_{ρπ}[f*∇logπ]. Lemma 4.2 only proves ∇J = E_{ρπ}[(ρhπ/ρπ)∇logπ] up to a positive scalar, via the variational derivative of KL. Replacing ρhπ/ρπ by the Kantorovich potential f* requires a separate argument that this substitution preserves, or usefully approximates, the policy-gradient direction. No such argument or bias bound is given. In fact, W1(ρπ,ρhπ) is a different functional from KL(ρhπ∥ρπ) (and from J), so its derivative w.r.t. ρπ being -f* does not imply E_{ρπ}[f*∇logπ] is a valid ascent direction for J. Lemma 4.4 only shows that any 1-Lipschitz f has variance ≤D²/4 — a property shared by all 1-Lipschitz functions, so it does not validate f* as a useful credit-assignment signal. The offline variant compounds this: ρh is fixed from 8 initial-policy successes, so even the on-policy identity in Lemma 4.2 does not apply. Thus the variance bound, while true, is not sufficient to support the claim that HPO's update improves expected return; the method currently rests on empirical support plus a heuristic bias–variance trade-off.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Hindsight Policy Optimization (HPO), a policy-gradient method for long-horizon language-agent RL. It defines a hindsight distribution of successful state-action pairs, projects these into a semantic embedding space, and uses the Kantorovich potential of the 1-Wasserstein distance between the current and hindsight distributions as a step-level advantage signal. This is combined with a GRPO-style episode-level advantage. The authors claim that aggregating semantically similar actions in intent space yields a bounded-variance estimator and more stable training. Experiments on SearchQA and TextCraft with Qwen2.5-3B/7B show consistent improvements over PPO, GRPO, Search-R1, and ZeroSearch, and ablations indicate that the step-level advantage alone can drive learning. The paper also reports negligible computational overhead for the added Wasserstein computation.","tokens_in":21634,"tokens_out":5414,"duration_ms":50348,"significance":"If the theoretical claims were valid, HPO would be a notable contribution: a critic-free, semantically aware estimator that reduces gradient variance in long-horizon LLM agent training while adding under 1% overhead. The empirical results are strong and well supported: multiple benchmarks, multiple seeds, ablations on the step-level weight, embedding-model robustness, and an interpretability case study. The paper also provides a public code link. However, the theoretical foundation has serious gaps: the key policy-gradient identity is asserted without proof, the variance bound is generic and not tied to the actual estimator, and the offline variant suffers from an unquantified distribution-shift bias. The central claim therefore currently rests on empirical evidence plus a heuristic bias-variance argument.","major_comments":[{"comment":"Equation (14) asserts ∇θJ(θ) = E_{ρπ}[f* ∇logπ] with f* the Kantorovich potential of W1(ρπ, ρhπ), but Lemma 4.2 only derives this form for the KL divergence, where the signal is the pointwise ratio ρhπ/ρπ. W1 is a different functional; its derivative with respect to ρπ does not automatically yield the policy gradient of J. No proof or bias bound is given that E[f*∇logπ] is a valid ascent direction. Moreover, the sign conventions conflict: Eq. (14) defines δW1/δρπ = f*, while Lemma 4.4 defines f* = −δW1/δρπ, and Eq. (16) uses AS = (−f* − mean)/std. The case study in §E.2 assigns positive AS to the final correct answer and negative AS to an uninformative step, which is reversed if f* is the usual Kantorovich potential for W1(ρπ, ρhπ). This sign inconsistency makes the actual updated quantity ambiguous and needs resolution.","section":"§4.3, Eq. (14); §4.4, Lemma 4.4"},{"comment":"The bound Varρπ(f*) ≤ D²/4 holds for any 1-Lipschitz function on the support of ρπ, not specifically for the Kantorovich potential. It therefore does not validate f* as a useful credit-assignment signal. Furthermore, the bound is on the population variance of f* under ρπ, whereas Algorithm 1 uses an empirical distributions constructed from G rollouts (and 8 offline trajectories for HPOoff). The variance of the finite-sample estimator is not analyzed. The paper's claim of a 'bounded-variance estimator' is thus not actually established. The authors should state the estimator whose variance is bounded and provide a finite-sample bound, or weaken the claim.","section":"§4.4, Lemma 4.4"},{"comment":"The hindsight distribution is defined theoretically as ρhπ = ρπ Qπ / Z, which depends on the current policy π. In offline HPO, ρhπ is instead constructed once from a fixed set of 8 successful trajectories generated by the initial policy. As training changes the policy, this fixed distribution is not recomputed against the current Qπ, so the step-level advantage is a biased estimator of the intended hindsight signal. No bound on this distribution-shift bias is provided. Since the main results (HPOoff) rely on this offline construction, the central theoretical justification does not apply to the primary evaluation setting.","section":"§4.2, Eq. (12); Appendix B; Algorithm 1"},{"comment":"The occupancy measures in Eq. (12) are defined as sums over time of Dirac measures without a normalizing factor, yet the Wasserstein distance in Eq. (13) requires probability measures. With γ = 1, the (1−γ) factor in §2.2 vanishes, so the measures are unnormalized. The paper needs to clarify how ρπ and ρhπ are normalized before computing W1, or the theoretical statements are not well-defined.","section":"§2.2, Eq. (12); §4.2"}],"minor_comments":[{"comment":"Typo in the proposition title: 'estimatior' should be 'estimator'.","section":"§4.3, Proposition 4.3"},{"comment":"Typo: 'search senario' should be 'search scenario'.","section":"Appendix A.1"},{"comment":"The case study is repeated three times ('Example 1: the advantage of GRPO', 'Example 1: the advantage of PPO', 'Example 1: the advantage of HPO') with identical trajectories. This appears to be an editorial duplication; only the HPO version is relevant to the main text.","section":"§E.2"},{"comment":"The proof of Lemma 3.1 invokes a 'standard simplifying assumption' that g(a) and Q̂(s,a) are conditionally independent given s, but this assumption is not stated in the lemma. It should be stated explicitly in the main text, since the variance decomposition does not hold in full generality.","section":"§D.1, Lemma 3.1 Proof"},{"comment":"The normalization in Eq. (16) uses std(f*) in the denominator but subtracts mean(−f*) in the numerator; this is algebraically equivalent to normalizing −f*. Please clarify the intended sign convention and ensure consistency with Eq. (14).","section":"§4.3, Eq. (16)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical component, but the theoretical section is not yet rigorous. The sign inconsistency in the definition of f* and the unproven Eq. (14) are the most serious issues. I would recommend asking the authors to either provide a rigorous derivation or a bound on the bias introduced by using the Wasserstein potential, or to reframe the paper's contribution as an empirically motivated heuristic with a weaker theoretical claim. The distribution-shift issue in offline HPO also needs to be addressed, since the main results use the offline variant."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely new method — comparing the current policy and a hindsight distribution in a semantic embedding space via Wasserstein distance and using the Kantorovich potential as a step-level advantage — and it performs well across a broad set of long-horizon agent benchmarks. The offline variant, which keeps a fixed hindsight set from initial-policy successes, is a practical idea and appears to stabilize training with negligible overhead. The paper includes code, detailed appendices, ablations, and a useful interpretability case study. It deserves a serious referee.\n\nThe theoretical story, though, is weaker than the presentation. Lemma 4.2 is a known identity: the policy gradient is proportional to E[rho_h/rho_pi * grad log pi]. Eq. (14) then asserts, without proof, that replacing that ratio by the Kantorovich potential f* gives a valid policy gradient. W1 is a different functional from KL and from J; the bounded variance of f* in Lemma 4.4 holds for any 1-Lipschitz function, so it does not establish that this biased signal points in an ascent direction. The stress-test note is right about this. The sign conventions in Eqs. (13)-(16) also do not line up with the case-study values as stated; that is fixable but needs cleanup.\n\nThe bigger practical problem is empirical. Appendix A.2 says they save a checkpoint every 50 steps and report the best testing performance. With a 4-7% claimed gain over baselines, best-checkpoint-on-test selection can plausibly account for most of the improvement, and the standard deviations in the offline rows are suspiciously small. They need to report fixed last-checkpoint or validation-selected numbers, and include checkpoint-selection variability in the error bars.\n\nThe offline variant also departs from the on-policy theory without a bias bound. A fixed hindsight distribution from 8 trajectories per query, generated by the initial policy, is not reweighted by the current Qpi as training changes the policy. The variance comparison is about the population on-policy object; neither the finite-sample estimator nor the distribution-shift bias is analyzed. This does not make the method useless — the empirical results suggest the signal carries real information — but the paper currently rests on heuristics plus experiments, not on the stated theory.\n\nWho should read it: anyone working on credit assignment or RLVR for multi-turn LLM agents. I would send it to review, but with a request for major revision. My own verdict is skeptical until the gradient claim is either proved, weakened to a heuristic, or supported by a bias bound, and until the checkpoint-selection issue is fixed.","headline":"A genuinely new credit-assignment idea for LLM agents, with promising results, but the theory overreaches at Eq. (14) and the main experiments are inflated by best-checkpoint-on-test reporting.","tokens_in":22088,"tokens_out":5336,"would_cite":true,"duration_ms":52267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hindsight Policy Optimization claims that long-horizon language-agent RL instability comes from pointwise comparisons in a huge discrete action space, and that an intent-space Wasserstein comparison yields a bounded-variance, critic-free si","keywords":["hindsight policy optimization","long-horizon reinforcement learning","language agents","Wasserstein distance","intent space","variance reduction","credit assignment","LLM training"],"falsifier":"Train HPO on a task where per-step causal credit is known by counterfactual action replacement, and check whether the step-level advantage ranks actions in the same order as the ground-truth causal effect; disagreement would show that bounded variance was achieved at the price of unacceptably biased credit assignment. A second check: keep the embedding fixed and measure the empirical variance of the Kantorovich potential across batches as the horizon grows; if it grows without bound despite a fixed embedding diameter, the assumptions behind the variance bound are violated.","tokens_in":21152,"feed_emoji":"🎯","tokens_out":4972,"duration_ms":44135,"temperature":0.7,"pith_summary":"The paper tries to establish that the instability of reinforcement learning for long-horizon language agents comes from comparing the current policy and the hindsight (successful-trajectory) distribution pointwise in a vast discrete language action space. It introduces Hindsight Policy Optimization (HPO), which embeds state-action pairs into a semantic intent space and uses the Wasserstein distance between the two distributions as the learning signal. The paper argues that this shift turns an estimator whose variance can be arbitrarily large into one bounded by the embedding-space diameter, and that this stabilizes and accelerates training without a critic. Experiments on search QA and a text-based game environment report consistent gains and show that the step-level signal can drive learning even when outcome rewards are removed.","feed_headline":"Wasserstein hindsight signal stabilizes long-horizon language agents","feed_subtitle":"Projecting policies and successful trajectories into intent space bounds gradient variance, preventing reward collapse.","key_machinery":"The central object is the hindsight occupancy measure ρhπ(s,a)=ρπ(s,a)Qπ(s,a)/Z, the distribution over state-action pairs that would be obtained by resampling actions after observing a successful outcome. HPO compares ρhπ with the current occupancy ρπ through the 1-Wasserstein distance W1(ρπ, ρhπ) on the intent embedding space; the optimal dual function f* — the Kantorovich potential — provides the step-level advantage. Its 1-Lipschitz property is what yields the bounded variance, allowing semantic neighbours to share statistical evidence instead of being treated as isolated atoms.","core_discovery":"This paper establishes that a policy gradient estimator can be built by treating 'what the agent would have done, knowing the final outcome' (the hindsight distribution) as the target, then measuring the Wasserstein distance in an intent space between the current occupancy and that hindsight occupancy. The dual Kantorovich potential serves as a per-step advantage. Because this potential is 1-Lipschitz, its variance under the current policy is bounded by D²/4, where D is the diameter of the embedded state-action support; the KL-based pointwise ratio signal has variance equal to a chi-squared divergence that can be arbitrarily large. HPO therefore replaces the unbounded pointwise ratio with a","pith_inferences":["If the hindsight distribution were periodically refreshed from the current policy's successful rollouts, the distribution-shift bias in offline HPO might shrink; the paper's online/offline comparison suggests a middle ground that combines stability with adaptivity, but the paper does not test it.","The same intent-space Wasserstein argument should transfer to dense or continuous rewards via the reweighted hindsight definition in Appendix B, and to multimodal agents if a suitable semantic embedding exists; these extensions are left untested.","HPO's practical guarantees depend on the quality of the semantic encoder; the reported encoder-ablation rank correlations show consistency across encoders, but rank alignment does not by itself establish that the resulting advantages are correct credit assignment.","A natural extension would use HPO's step-level advantage as a dense reward for a critic-based method, potentially combining the variance reduction of both families; this is not explored in the paper."],"forward_implications":["Long-horizon language-agent training can be stabilized without training a critic, avoiding the slow warm-up of PPO while retaining step-level credit assignment.","The variance bound D²/4 means the learning signal degrades gracefully with the semantic diameter of the intent embedding, so better embedding spaces directly improve signal-to-noise ratio.","Because the step-level advantage comes from mass transport between distributions, statistical evidence is shared across semantically similar surface forms, improving sample efficiency in sparse-reward settings.","HPO's step-level advantage alone is sufficient to drive policy improvement, offering a route to learning from offline data that has already been evaluated without requiring new outcome rewards.","In the evaluated long-horizon tasks, HPO improves average performance by 4–7 points over PPO and GRPO while adding less than 1% per-iteration overhead."],"fun_headline_variants":["Hindsight in intent space cuts RL variance for long-horizon agents","Bounded-variance RL via Wasserstein hindsight in intent space","HPO: Wasserstein hindsight bounds variance for long-horizon RL","Outcome-aware policy gradient with bounded variance via hindsight"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The theoretical hindsight distribution is defined relative to the current policy, but offline HPO constructs it once from a fixed set of eight successful trajectories per query collected by the initial policy and never re-estimates it as training changes the policy, so the step-level advantage can be a biased estimate of the intended hindsight signal with no bound on that distribution-shift bias.","fun_headline_variants_meta":{"raw":{"variants":["Hindsight in intent space cuts RL variance for long-horizon agents","Bounded-variance RL via Wasserstein hindsight in intent space","HPO: Wasserstein hindsight bounds variance for long-horizon RL","Outcome-aware policy gradient with bounded variance via hindsight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3241,"prompt_tokens":650,"completion_tokens":2591,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":2518}},"tokens_in":394,"tokens_out":2591,"duration_ms":16462,"temperature":1.0,"reasoning_tokens":2518,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:40:45.427615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HPO on a task where per-step causal credit is known by counterfactual action replacement, and check whether the step-level advantage ranks actions in the same order as the ground-truth causal effect; disagreement would show that bounded variance was achieved at the price of unacceptably biased credit assignment. A second check: keep the embedding fixed and measure the empirical variance of the Kantorovich potential across batches as the horizon grows; if it grows without bound despite a fixed embedding diameter, the assumptions behind the variance bound are violated.","supporting_citations":[],"review_version":1}