{"id":"88a248a6-5224-44af-a9bf-dcdb1ce8477d","arxiv_id":"2507.08537","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Recursive reward aggregation defines Bellman equations for any objective computable by folding rewards, enabling RL agents to optimize max, min, mean, variance, and Sharpe ratio without reward redesign.","lead":"This paper proposes an alternative to reinforcement learning's usual trick of adding up rewards: instead, the agent can combine rewards with rules like 'take the maximum', 'take the minimum', or 'compute the mean', all within the same learning machinery. The authors show that many such rules lead to valid learning updates, and they demonstrate the approach on grid-world, control, and financial portfolio tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.8's Bellman optimality equation is false for non-monotone aggregations; a concrete variance/mean counterexample shows the missing order-preservation assumption is load-bearing.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap: Theorem 3.8 requires an order-preservation/monotonicity condition that is absent, and the theorem is not satisfied by non-monotone aggregations such as variance and mean. My stress-test confirms this with a concrete deterministic counterexample in which the Bellman optimality equation (Eq. 12) is numerically false: the recursively computed statistic using the optimal continuation from the next state differs from the true optimal statistic from the current state. The counterexample also shows a stronger pathology: no single policy is optimal for all states under a non-monotone objective, so Definition 3.7's globally optimal policy may not exist. This makes the central claim about generalizing the Bellman optimality equation to arbitrary recursive aggregations, and hence the value-based optimization of Sharpe ratio and variance, unsupported as stated. The policy-evaluation side (Theorem 3.2) and the empirical demonstrations of distinct behaviors under different aggregations remain valid; the paper is a useful framework for monotone aggregations and could be repaired by adding an explicit monotonicity assumption and restricting the optimality claims. Since the reader already reached CONDITIONAL on this same weakness, my finding does not move the verdict.","tokens_in":51626,"tokens_out":18525,"duration_ms":221002,"concrete_test":"Implement the deterministic MDP: s→u with reward 0; u→v1 with reward 100, then v1→T with reward −100; u→v2 with reward 1000, with v2 terminal. Use variance aggregation with init (0,0,0), update (n,s,q)↦(n+1,s+r,q+r²), post=q/n−(s/n)², and lower-variance-better. Enumerate the two stationary policies (choice at u) to compute τ*(s)=variance([0,100,−100])=6666.7 and τ*(u)=variance([1000])=0. Then evaluate the RHS of Eq. (12) at s: 0▷τ*(u)=variance([0,1000])=250000. If these differ, Theorem 3.8 is false. Repeat with mean aggregation (post=s/n, higher-mean-better) using suffixes that make the same non-monotonicity appear.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 3.8 justifies the key step by 'pointwise maximization' after writing τ*(s) = r(s,π*(s)) ▷ τ*(p(s,π*(s))). This step requires that the action chosen by a globally optimal policy at s also maximizes the expression r(s,a) ▷ τ*(p(s,a)), which is equivalent to requiring that the update function r ▷ (−) is monotone with respect to the preorder ≤T used to define optimality. That monotonicity is not stated and fails for several aggregations in Table 1, including variance, mean, range, and Sharpe ratio. Concretely, let aggregation be variance (lower is better) and let states s,u,v1,v2 with: s→u, reward 0; u→v1, reward 100, then v1→T, reward −100; u→v2, reward 1000, then v2 terminal. At u, suffix A=[100,−100] has variance 10000 and suffix B=[1000] has variance 0, so B is optimal at u. From s, however, full trajectories are [0,100,−100] with variance 6666.7 and [0,1000] with variance 250000, so A is optimal at s. Thus no single stationary policy is optimal for both s and u, and Eq. (12) fails: the RHS using τ*(u)=B gives 250000, not the true optimal statistic 6666.7. The same failure occurs for mean aggregation with an analogous prefix. This invalidates the claimed extension of value-based dynamic programming (Q-learning, DQN) to non-monotone recursive aggregations, directly undermining the central claim that value-based algorithms can optimize any recursive aggregation, including Sharpe ratio, without state augmentation. Policy evaluation (Theorem 3.2) remains sound; the gap is specifically the Bellman optimality equation and the definition of a globally optimal policy in Definition 3.7, which may not exist for such objectives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an algebraic framework for reinforcement learning in which the standard discounted-sum return is replaced by a recursive reward aggregation defined by an initial statistic init, an update function ▷, and a post-processing function post. The central claim is that composing a recursively generated reward sequence with such an aggregation yields a generalized Bellman equation for the state statistic function (Theorem 3.2), that fixed points of the associated Bellman operator are unique under a contractiveness condition (Theorem 3.6), and that an analogous Bellman optimality equation holds for optimal policies (Theorem 3.8). The authors extend the framework to state-action statistics, stochastic transitions, and actor-critic and value-based algorithms, and they report experiments in grid-world, continuous control, and portfolio optimization, including optimization of discounted max, min, mean, variance-regularized sum, and Sharpe ratio objectives.","tokens_in":51987,"tokens_out":7046,"duration_ms":86260,"significance":"The policy-evaluation side of the paper is elegant and potentially useful: the fusion perspective gives a clean derivation of a generalized Bellman equation for a large class of recursive aggregations, and the fixed-point uniqueness result is a natural generalization of the classical contraction argument. The paper also makes a welcome effort to provide explicit recursive formulations, a table of aggregations, and empirical validation, all of which contribute to reproducibility. However, the policy-optimization claim is not supported as stated: Theorem 3.8's Bellman optimality equation is false for non-monotone aggregations such as variance, mean, range, and Sharpe ratio, which are precisely among the paper's advertised examples. This is a load-bearing gap because value-based algorithms (Q-learning, DQN) and the grid-world experiments rely on that theorem. The stochastic/portfolio claims are also weaker than stated because the PPO implementation optimizes aggregated expected rewards rather than expected aggregated rewards.","major_comments":[{"comment":"The step labelled 'pointwise maximization' in the proof (the line marked as (99) in Appendix F) is valid only if the update function satisfies a monotonicity condition with respect to the preorder ≤T: for every reward r, τ1 ≤T τ2 must imply r ▷ τ1 ≤T r ▷ τ2. This assumption is not stated in Theorem 3.8, and it fails for several aggregations in Table 1, including variance, mean, range, and Sharpe ratio. Concretely, consider the variance aggregation with lower variance preferred, so that sup in the preorder means minimum variance. Let states s, u, v1, v2 with transitions s→u (reward 0), u→v1 (reward 100) then v1→T (reward −100), and u→v2 (reward 1000) then v2 terminal. From u, the suffixes [100,−100] and [1000] have variances 10,000 and 0, respectively, so [1000] is better at u; from s, the full trajectories [0,100,−100] and [0,1000] have variances 6,666.7 and 250,000, respectively, so [100,−100] is better at s. Hence no stationary policy is optimal at both states, τ* is not well-defined, and Eq. (12) fails because the right-hand side using the optimal statistic at u gives variance 250,000 rather than the true optimal value 6,666.7. The theorem can be repaired by adding the missing monotonicity assumption and restricting the value-based claims to aggregations that satisfy it, but as stated the theorem is false.","section":"Section 3.3, Theorem 3.8 and Eq. (12)"},{"comment":"The stochastic and portfolio claims are weaker than the text suggests. Equation (13) is a distributional recursion for the distribution of the aggregated statistic, but the PPO implementation in Algorithm 2 uses a point-valued critic and sampled transitions, which computes the statistic of expected rewards (aggregated expected rewards) rather than the expected aggregated statistic. The paper itself acknowledges in Appendix H.2 that this differs from the expected aggregated rewards. Therefore the statement in Section 5.4 that the method estimates and optimizes the exact Sharpe ratio over the entire trajectory is not justified for the stochastic PPO setting: expectation does not commute with the Sharpe ratio, so the optimized objective is not the expected Sharpe ratio. The deterministic policy-evaluation results are unaffected, but the stochastic and portfolio claims need to be reworded and the experiments re-described accordingly.","section":"Section 4, Appendix H.2, and Section 5.4"},{"comment":"The Q-learning update τ(s,a) ← τ(s,a) + α(max_{a′} r ▷ τ(s′,a′) − τ(s,a)) is not well-defined for most of the statistics in Table 1. For the mean aggregation, T = N × R, so τ(s,a) is a pair (length, sum) and the expression τ(s,a) + α(·) is undefined; similar problems arise for variance, range, and top-k buffers. The paper does not specify the vector-space structure on T or an alternative update for these cases. As written, Algorithm 1 supports only aggregations whose statistic space is a vector space, such as the discounted sum. The grid-world experiments in Section 5.1 therefore need either a concrete definition of the statistic space and the addition operation used in the implementation, or a different update rule.","section":"Appendix G.1, Algorithm 1"}],"minor_comments":[{"comment":"The theorem statement says the state-action statistic function has type τ S×Aπ : S → T, but the equation below it and the surrounding text use S × A → T; the domain should be S × A.","section":"Appendix A.3, Theorem A.1"},{"comment":"There is a typo in the definition: 'a recursive statistic aggregation function agginit,▷ : [R] → T of is defined' should read 'is defined'.","section":"Definition 3.1"},{"comment":"The proof of Lemma D.7 uses arg sup_{a∈A} f(a), which need not be unique; the argument can be repaired by choosing witnesses or by proving the two inequalities directly without naming an argmax.","section":"Appendix F, Lemma D.7"},{"comment":"In Eq. (13), the notation r ▷ τ′ is used with r and τ′ as random variables, but the joint distribution and measurability conditions on the stochastic update are not specified; please define the pushforward of the product measure explicitly.","section":"Section 4, Eq. (13)"},{"comment":"The Sharpe ratio is advertised as a key example, but Table 1 does not contain an explicit Sharpe-ratio row; the paper only says that mean and variance are recursive. Adding the explicit statistic, update function, and post-processing for the Sharpe ratio would make the example precise.","section":"Table 1 and Section 5.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea is genuinely attractive: view reward aggregation as a fold over the generated reward sequence, and the Bellman equation for policy evaluation falls out of the fusion law. That part is clean and correct. Theorem 3.2 and the fixed-point uniqueness under contractiveness are solid, and the table of aggregations (sum, max, min, mean, variance, range, top-k) is a useful catalogue. The experiments, especially the portfolio application, make the case that choosing a different aggregation changes behavior in practice.\n\nThe problem is Theorem 3.8. The proof's \"pointwise maximization\" step silently assumes the update function is monotone with respect to the preorder on statistics. That is true for max, min, and mean, but it is false for variance, range, and Sharpe ratio. The stress-test counterexample is correct: with rewards [0,100,-100] vs [0,1000] the variance-optimal policy is not greedy with respect to the optimal suffix variance, so the Bellman optimality equation fails. The claim that Q-learning or DQN can optimize any recursive aggregation without state augmentation does not hold. The paper even notes that Cui & Yu (2023) relies on order-preserving updates, so the missing assumption was already known in the literature.\n\nThis is not a minor technicality: it cuts the advertised scope roughly in half. The fix is to state monotonicity explicitly, or to separate aggregations that satisfy it (where value-based methods work) from those that do not (where the correct approach is state augmentation, as in Veviurko et al., or distributional RL). The stochastic-setting discussion is honest but thin: the paper acknowledges that expected aggregated rewards differ from aggregated expected rewards and defers convergence to future work, which is acceptable but means the \"applies to stochastic settings\" claim in the abstract is mostly about definitions, not guarantees.\n\nThe portfolio result (Sharpe 1.12 ± 0.92 vs 0.48 ± 0.79) is suggestive, not conclusive, given overlapping standard deviations. Still, the methodology is a reasonable way to directly optimize a non-cumulative objective.\n\nBottom line: the policy-evaluation half is worth keeping; the policy-optimization half needs a substantial caveat. A good referee could guide the authors to a publishable revision. I'd send it out, but I'd warn the editor that the main theorem currently overclaims.","headline":"Elegant fold-based framework for policy evaluation, but the Bellman optimality theorem fails for non-monotone aggregations like variance; the value-based optimization claims are overbroad.","tokens_in":52599,"tokens_out":4635,"would_cite":true,"duration_ms":50705,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that the standard discounted-sum Bellman equations generalize to any recursively computed reward aggregation—max, min, mean, variance, and Sharpe ratio—so RL agents can optimize these objectives directly without touching…","keywords":["recursive reward aggregation","Bellman equation","algebra fusion","dynamic programming","Sharpe ratio","distributional reinforcement learning","Markov decision process"],"falsifier":"In a two-step MDP where the objective is variance (or a variance-penalized sum), compute the optimal trajectory by exhaustive enumeration and compare it with the policy produced by the Bellman optimality update $\\tau_*(s) = \\sup_a (r(s,a) \\triangleright \\tau_*(p(s,a)))$ using the paper's variance update; different outcomes would show Theorem 3.8 does not hold for that aggregation.","tokens_in":51402,"feed_emoji":"📈","tokens_out":6425,"duration_ms":66499,"temperature":0.7,"pith_summary":"This paper claims that the Bellman equations of reinforcement learning are not special to the discounted sum of rewards: they hold for any objective that can be computed by folding a reward sequence with a fixed initial value and update rule. The authors introduce \"recursive statistic aggregation,\" in which a statistic (a single number or a small vector) is updated one reward at a time, and show that composing this fold with the recursive generation of rewards yields a generalized Bellman equation. That means value-based and actor-critic algorithms can directly optimize objectives such as discounted max, discounted min, mean, range, variance, and the Sharpe ratio, without changing the reward function or enlarging the state space. If correct, this shifts goal specification in RL from \"what to reward\" to \"how to evaluate reward sequences,\" a cheaper and more flexible design axis.","feed_headline":"Bellman equations hold for max, mean, and Sharpe ratio too","feed_subtitle":"One recursive fold turns any such objective into a standard RL update, no reward redesign needed.","key_machinery":"The central object is the recursive statistic aggregation $\\mathrm{agg}_{\\mathrm{init},\\triangleright}$: a fold over reward lists defined by an initial statistic $\\mathrm{init} \\in T$, an update function $\\triangleright : R \\times T \\to T$, and a post-processing function $\\mathrm{post} : T \\to R$ that extracts a scalar value (Definition 3.1). The machinery that carries the argument is algebra fusion: composing the coalgebra that generates rewards one step at a time (the step function $\\mathrm{step}_\\pi$) with this fold as a catamorphism collapses the two recursions into one, yielding the Bellman equation $\\tau_\\pi(s) = r_\\pi(s) \\triangleright \\tau_\\pi(p_\\pi(s))$ for the state statistic function. This single recursive identity lets a Q-learning, PPO, or TD3 update target compute any listed aggregation by replacing the scalar \"discounted addition\" with the appropriate update rule.","core_discovery":"The central claim is that the Bellman equation is a fusion of two recursive structures: an MDP generates a reward list recursively through a step function, and a recursive statistic aggregation consumes that list through an update function $\\triangleright$ and an initial statistic. For any such aggregation (Definition 3.1), the composed state statistic function $\\tau_\\pi$ satisfies $\\tau_\\pi(s) = \\mathrm{init}$ at terminal states and $\\tau_\\pi(s) = r_\\pi(s) \\triangleright \\tau_\\pi(p_\\pi(s))$ elsewhere (Theorem 3.2), and the analogous optimality equation holds with a supremum over actions (Theorem 3.8). This reproduces the standard discounted sum while also covering discounted max, discounted min, log-sum-exp, range, mean, variance, and top-k, each with an explicit update rule in Table 1; the Sharpe ratio follows because it is post-processed from recursively maintained mean and variance statistics. The paper further shows the generalized Bellman operator has unique fixed points when the update function is contractive (Theorem 3.6), and it extends the recursion to stochastic transitions as a distributional Bellman equation.","pith_inferences":["A direct test of Theorem 3.8's scope would compare the policy from the Bellman optimality update against an exhaustive trajectory search in a small MDP for a non-monotone objective like variance; a mismatch would show the pointwise-maximization step requires an order-preservation assumption the paper does not state.","The framework opens a design axis where the aggregation function itself could be learned from data (e.g., a parameterized $\\triangleright$), rather than hand-picked, which the paper lists only as future work.","Non-recursive objectives like the median and CVaR fall outside the exact framework (the paper says so); sketching approximations might still combine with the same Bellman structure, but the per-step approximation error would need to be controlled.","The same fold-and-fuse lens applies beyond rewards to any per-step quantity—costs, constraint violations, or information signals—potentially unifying risk measures and constrained control within one Bellman-style recursion."],"forward_implications":["Value-based algorithms (Q-learning, DQN) and actor-critic algorithms (PPO, TD3) can optimize any objective expressible as a recursive aggregation by swapping the Bellman update, with no change to state space or reward function.","Objectives that previously required state augmentation or approximate reward shaping—discounted max, worst-case min, mean, range, and variance-regularized sums—become direct training objectives with the same per-step computational cost.","The Sharpe ratio can be optimized exactly, rather than through an approximate differential surrogate, by maintaining mean and variance statistics recursively; the portfolio experiment suggests this improves test Sharpe ratio.","In stochastic settings, the recursion yields a distributional Bellman equation for any recursive aggregation, so distributional RL methods extend beyond discounted sums.","The contractive-update condition (Theorem 3.6) guarantees uniqueness of the fixed point of the generalized Bellman operator, supporting iterative policy evaluation."],"supporting_citations":[{"why":"Supplies the algebra fusion law that collapses the reward-generation coalgebra and the aggregation fold into one recursion, producing the Bellman equation.","marker":"Hinze et al., 2010"},{"why":"Defines catamorphisms/folds over lists, the recursion scheme that formalizes recursive statistic aggregation.","marker":"Meijer et al., 1991"},{"why":"Provides the original dynamic-programming Bellman equations that the paper generalizes.","marker":"Bellman, 1966"},{"why":"Prior non-cumulative objective method that the paper identifies as a special case of its framework with an order-preserving update function.","marker":"Cui & Yu, 2023"},{"why":"Maximum-reward formulation that showed incorrect interchange of expectation and maximum, motivating the paper's recursive-statistic treatment.","marker":"Gottipati et al., 2020"},{"why":"State-augmentation approach to maximum objectives that the recursive framework avoids.","marker":"Veviurko et al., 2024"},{"why":"Source of the differential Sharpe ratio baseline in the portfolio experiment.","marker":"Moody et al., 1998"},{"why":"Non-cumulative MDP method for the Sharpe ratio, used as a baseline in the portfolio experiment.","marker":"Nägele et al., 2024"}],"fun_headline_variants":["One reward fold fits all: Bellman for max, mean, and Sharpe","Bellman without reward redesign: aggregate recursively instead","All your objectives are Bellman's: max, mean, variance, Sharpe","One recursive fold to rule all reward aggregations","Bellman is just a recursive aggregate: extend to max, Sharpe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Picking the best-looking next step at every state is assumed to add up to the best whole trajectory, but that holds only when the update rule and the way statistics are compared are aligned (order-preserving), which is not guaranteed for objectives like variance.","fun_headline_variants_meta":{"raw":{"variants":["One reward fold fits all: Bellman for max, mean, and Sharpe","Bellman without reward redesign: aggregate recursively instead","All your objectives are Bellman's: max, mean, variance, Sharpe","One recursive fold to rule all reward aggregations","Bellman is just a recursive aggregate: extend to max, Sharpe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2603,"prompt_tokens":907,"completion_tokens":1696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1609}},"tokens_in":523,"tokens_out":1696,"duration_ms":12485,"temperature":1.0,"reasoning_tokens":1609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:17:10.049994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a two-step MDP where the objective is variance (or a variance-penalized sum), compute the optimal trajectory by exhaustive enumeration and compare it with the policy produced by the Bellman optimality update $\\tau_*(s) = \\sup_a (r(s,a) \\triangleright \\tau_*(p(s,a)))$ using the paper's variance update; different outcomes would show Theorem 3.8 does not hold for that aggregation.","supporting_citations":[{"cited_title":"To the max: Reinventing reward in reinforcement learning","cited_arxiv_id":null,"evidence_quote":"State-augmentation approach to maximum objectives that the recursive framework avoids."}],"review_version":1}