{"id":"16295477-88b2-46f8-b72c-07650a9971ea","arxiv_id":"2502.09724","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An α-approximate portfolio of RL policies can cover all p-mean social welfare objectives for p ≤ 1 with size O(log κ / log(1/α)), and the paper gives algorithms and experiments for this.","lead":"The paper constructs small 'portfolios' of reinforcement learning policies that stay near-optimal across a family of fairness objectives called generalized p-means. It provides algorithms with theoretical size and oracle-complexity bounds, a cheaper heuristic, and experiments on taxi, disaster response, and healthcare tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1 for ℓ=1 rests on an invalid interchange in Lemma A.8 Part 2; the slope bound is likely repairable, but as submitted the ESR guarantees are unproven.","rationale":"I agree with the reader's CONDITIONAL verdict, but I locate the load-bearing risk differently. The reader's weakest_assumption emphasizes the exact-oracle and strictly-positive-reward assumptions. Those are genuine scope limitations: the healthcare and taxi experiments use Whittle-index heuristics, Welfare Q-Learning, and zero-valued rewards, so they do not instantiate the theorem's regime, and the reported 'actual approximation' is relative to algorithm-found values rather than v*. However, a theorem may correctly state conditions under which an oracle-based algorithm has guarantees even when the experiments only suggest practical behavior. The sharper problem is internal to the proof: Lemma A.8 Part 2 applies an invalid interchange of expectation and log-derivative for ℓ=1. This is the step that yields the slope bound on ln v*(p), without which the line-search iteration bound, the portfolio-size bound, and the warm-start proposition all lose support for ESR. The inequality itself is very likely true, because replacing the invalid step by the weighted-average formula and using 0 ≤ d/dp ln f ≤ κ ln κ gives the same bound, so I do not believe the theorem is false. But the manuscript as submitted does not contain a valid proof for one of the two aggregation rules it claims to cover, which is exactly the kind of repairable gap that warrants a conditional rather than an acceptance verdict. I also note the Table 3 inconsistency (Resource Allocation size 4 shows p-MEAN-PORTFOLIO at 0.100 while Table 1 shows 1.000 for the same entry); that is a data-quality issue that should be corrected but does not affect the theorem. The proposed analytic re-derivation, with the numerical cross-check, would settle whether the slope bound actually holds; if it holds, the fix is routine and the conditional verdict should remain only until the corrected proof appears.","tokens_in":23556,"tokens_out":14129,"duration_ms":155340,"concrete_test":"Independently re-derive Lemma A.8 Part 2 for ℓ=1 using the correct identity d/dp ln v(1)(π,p) = Eτ[f(G(τ),p) * d/dp ln f(G(τ),p)] / Eτ[f(G(τ),p)], and verify that monotonicity of f in p (Lemma 3.2) plus the pointwise bound 0 ≤ d/dp ln f ≤ κ ln κ from Part 1 imply the weighted average is at most κ ln κ. As a numerical cross-check, take a two-trajectory distribution with returns (1,M) and (M,1) under varying probabilities and M∈{10,100}, and evaluate d/dp ln v(1) at p∈{-10,-1,-0.1,0.1,0.9}; if any computed value exceeds κ ln κ, Lemma A.8 Part 2 is false and Theorem 4.1 fails for ℓ=1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing defect is in Lemma A.8 Part 2 (Appendix A.1), which supplies the log-Lipschitz bound on w(p)=ln v*(p) used to bound LINE SEARCH iterations and hence both the portfolio-size and oracle-complexity statements of Theorem 4.1, as well as Proposition 4.2. For ℓ=1, v(1)(π,p)=Eτ[f(G(τ),p)], and the proof claims d/dp ln v(1)(π,p) = Eτ[d/dp ln f(G(τ),p)] by linearity of expectation. This interchange is invalid: the derivative of the log of an expectation is a weighted average, d/dp ln E[X] = E[X * d/dp ln X] / E[X], not E[d/dp ln X]. Since Theorem 4.1 explicitly covers ℓ∈{1,2}, the central approximation and complexity result is not established as written for the expected-scalarized-returns case. The gap is probably repairable: because f is monotone in p and the pointwise derivative of ln f is bounded by κ ln κ, the correct weighted-average formula should also give d/dp ln v(1)(π,p) ≤ κ ln κ. But the submitted proof does not supply that argument, and no other part of the paper covers ℓ=1. The exact-oracle and strictly-positive-reward assumptions are real limitations, especially because the taxi and healthcare experiments use approximate solvers and zero-valued rewards, but those are stated assumptions rather than internal contradictions; the Lemma A.8 proof gap is an internal correctness risk in the theorem itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies α-approximate portfolios for generalized p-mean social welfare functions in multi-objective RL. For p ≤ 1 and the two aggregation rules v(1) (expected scalarized returns) and v(2) (scalarized expected returns), it defines a portfolio as a set of policies Π′ such that for every p some policy in Π′ is within factor α of the optimal value v*(p). The main algorithm, p-MEAN PORTFOLIO (Algorithm 1), iteratively selects p values via a line-search subroutine (Algorithm 2), and Theorem 4.1 claims a portfolio of size O(ln κ / ln(1/α)) with oracle complexity eO((ln κ)^2 ln ln N / ln(1/α)), assuming an exact oracle for Problem (3) and strictly positive rewards with condition number κ. The paper also proposes a budget-constrained heuristic (Algorithm 3), proves a warm-start bound (Proposition 4.2), and reports experiments on a taxi domain, a synthetic disaster-resource-allocation domain, and an RMAB healthcare intervention domain.","tokens_in":23946,"tokens_out":9282,"duration_ms":95574,"significance":"If the main theorem were fully established, this would be a substantial contribution: it reduces a continuum of p-mean fairness objectives to a compact portfolio with explicit approximation guarantees, and it is framed uniformly for both ESR and SER aggregation. The paper also provides a useful empirical study, a public code repository, and a warm-start result that is of independent interest. The main proof, however, contains two incorrect inequalities in Lemma A.8 that are load-bearing for Theorem 4.1 and Proposition 4.2; both are plausibly repairable, but as submitted the formal guarantees for ℓ=1 are not demonstrated. In addition, the experimental section operates outside the theorem's stated assumptions (approximate oracles and zero rewards), so the current version should be revised before the guarantees are presented as established.","major_comments":[{"comment":"The proof claims that d/dp ln v(1)(π,p) = Eτ[d/dp ln f(G(τ),p)] by linearity of expectation. This is incorrect: the derivative of a log-expectation is a weighted expectation with weights proportional to f(G(τ),p), not the unweighted expectation. Since the pointwise derivative is bounded by κ ln κ, the inequality d ln v(1)/dp ≤ κ ln κ is recoverable by applying the weighted-average formula, but the submitted argument does not supply that derivation. This gap affects Theorem 4.1's portfolio-size and oracle-complexity guarantees as stated for ℓ=1, and it also affects Proposition 4.2.","section":"Appendix A.1, Lemma A.8 Part 2"},{"comment":"The displayed inequality in the proof of Part 3 has the wrong direction. Since w(q)=maxπ ln v(π,q) and πp maximizes at p, one has w(q) ≥ ln v(πp,q), so [w(q)−w(p)]/(q−p) is at least, not at most, [ln v(πp,q)−ln v(πp,p)]/(q−p). The slope bound on w can be recovered by a standard envelope argument, but the written proof does not establish the bound needed for the LINE SEARCH iteration count in the oracle-complexity analysis.","section":"Appendix A.1, Lemma A.8 Part 3"},{"comment":"Theorem 4.1 is stated under an exact oracle for Problem (3) and under Assumption 3.1 with L>0. The taxi environment assigns zero and negative rewards, the healthcare environment uses a 0/1 reward, and the healthcare and taxi experiments use approximate solvers (Whittle-index heuristic and Welfare Q-Learning). Moreover, Section 4 explicitly replaces v*(p) by the value found by the RL algorithm when measuring approximation. Consequently Tables 1–2 and Figure 1 do not operate in the theorem's regime and cannot be read as direct empirical verification of the α-guarantee. The authors should state this mismatch explicitly and clarify what is actually being measured in the experiments.","section":"Section 4 and Section 5 vs. Assumption 3.1"},{"comment":"The heuristic justifies the approximation u(l) by assuming that v(π_{pm(l)}, p)/v*(p) is monotonically decreasing on each interval [pm(l), pm(l+1)]. The text says this assumption 'holds exactly when the interval is sufficiently small, as this function is continuous in p (Lemma A.8) and attains its maximum at p=pm(l)'. Continuity plus endpoint maximality does not imply monotonicity, and Lemma A.8 does not supply this monotonicity. Since Algorithm 3 is presented as a heuristic this is not fatal, but the stated justification is not valid and should be corrected or explicitly weakened.","section":"Section 4.2, BUDGET-CONSTRAINED PORTFOLIO"}],"minor_comments":[{"comment":"There are typos in the normalization step: 'LH = 1' should describe the normalized total-reward bound, and 'ln R' in Case B should be 'ln κ' or a defined quantity.","section":"Appendix A.1, proof of Lemma A.8 Part 1"},{"comment":"The bound displayed in equation (8) is O((ln κ)^2 ln ln N / (ln(1/α) ln ln(1/α))), while Theorem 4.1 states eO((ln κ)^2 ln ln N / ln(1/α)); the eO notation should be defined precisely or the two displays reconciled.","section":"Appendix A.1, proof of Theorem 4.1"},{"comment":"The sentence 'We note that this size bound can be tightened slightly' appears to refer to the oracle-complexity bound rather than the portfolio-size bound.","section":"Appendix A.1, proof of Theorem 4.1"},{"comment":"The notation '[∞, 1]' appears where '[−∞, 1]' is intended, and the phrase 'partition the interval [∞, 1]' should be fixed.","section":"Section 4.2"},{"comment":"The Resource Allocation row for portfolio size 4 lists the p-MEAN-PORTFOLIO approximation as 0.100, which is inconsistent with Table 1's value of 1.000 for the same row and is likely a typo.","section":"Appendix C, Table 3"}],"recommendation":"major_revision","confidential_remarks":"I concur with the stress-test concern: Lemma A.8 Part 2 contains a genuinely invalid interchange, and Part 3 contains an additional inequality-direction error that the reader's report did not flag. Both are repairable locally, so I do not recommend rejection, but the paper should not be accepted with the current Appendix A. I also recommend requiring the authors to state clearly that the experimental setups do not satisfy the theorem's exact-oracle and strictly-positive-reward assumptions, and to separate heuristic/empirical claims from the formal guarantees."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful within-subfield paper with a plausible central theorem, a real proof gap in the ESR case, and experiments that don't quite live in the theorem's regime. Worth serious refereeing, but as submitted the guarantees for the expected-scalarized-returns case are unproven.\n\nWhat's new: this is the first portfolio construction for p-mean welfare objectives in multi-objective RL, covering both ESR and SER aggregation rules, with size bound O(ln kappa / ln(1/alpha)) and an oracle-complexity bound. The line-search-over-p idea and the alpha-approximate portfolio framing come from the same group's earlier EC and SODA work in combinatorial optimization; the extension to sequential decision-making, the warm-start bound, and the budget-constrained heuristic are genuine additions. The paper is honest about the oracle assumption in Section 4, and the code is released.\n\nWhere it is soft: the stress-test note is right. Lemma A.8 Part 2 for l=1 claims d/dp ln E_tau[f(G(tau),p)] = E_tau[d/dp ln f(G(tau),p)], which is not linearity of expectation. The correct formula is a weighted average, E[f' ]/E[f], and the submitted proof does not supply the needed bound. Since Theorem 4.1 covers l in {1,2} and Proposition 4.2 also uses the slope bound, the ESR guarantees are not established as written. The gap looks repairable: the pointwise slope bound is kappa ln kappa, and a weighted-average argument should give the same slope on ln v^(1). But it has to be written.\n\nThe exact-oracle and strictly-positive-reward assumptions create a second gap between theory and experiment. The theorem needs an exact solver for Problem (3); the healthcare experiments use a Whittle-index heuristic, and the taxi reward includes zeros, so kappa is undefined there. The paper acknowledges the exact-oracle issue in prose but does not reconcile it with the theorem statement. Minor: Table 1's caption says p-MEAN PORTFOLIO consistently outperforms, but in the taxi rows random p sampling ties at portfolio sizes 1 and 2; Table 3 has a 0.100 entry that looks like a typo. These should be cleaned up.\n\nWho it is for: people working on multi-objective RL, fairness in resource allocation, and decision-makers who want a small set of policies rather than committing to a single p. The citation pattern is self-referential but legitimate: the prior portfolio papers are the actual source of the technique. I would bring this to our reading group.\n\nRecommendation: send it to peer review. The core idea is likely sound and the application area is timely, but the referee should require a fixed l=1 slope proof and an explicit statement of when the oracle and positivity assumptions hold in practice.","headline":"Useful MORL portfolio paper with a plausible core idea, but the ESR guarantees currently rest on an invalid derivative interchange and the experiments don't satisfy the theorem's oracle or positivity assumptions.","tokens_in":24435,"tokens_out":2207,"would_cite":true,"duration_ms":24759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68W25","91B14","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A small set of policies can cover every fairness preference at once.","keywords":["multi-objective reinforcement learning","social welfare functions","generalized p-means","policy portfolio","approximation guarantees","fairness","oracle complexity","scalarized returns"],"falsifier":"Run p-MEAN PORTFOLIO on a small MDP with $N = 2$ rewards where exact optimal policies are enumerable, with fixed $L$, $U$ and $\\alpha = 0.5$, and test every $p$ in a fine grid: finding any $p \\leq 1$ where no portfolio policy reaches $\\alpha$ times the true optimum, or any instance where the returned portfolio exceeds the Theorem 4.1 size bound, would refute the theorem. A second test is to allow one reward to be zero, which makes $\\kappa$ undefined and shows whether the strictly-positive assumption is essential.","tokens_in":23372,"feed_emoji":"⚖️","tokens_out":11160,"duration_ms":98207,"temperature":0.7,"pith_summary":"This paper asks how a decision-maker can choose a policy when multiple stakeholders have different rewards and the right fairness criterion is not known in advance. It works with generalized $p$-mean social welfare functions, the family that interpolates between Egalitarian welfare at $p = -\\infty$, Nash welfare at $p = 0$, and Utilitarian welfare at $p = 1$, and it asks whether a small set of policies can be approximately optimal for every $p \\leq 1$ simultaneously. The main result is an algorithm, p-MEAN PORTFOLIO, that constructs such an $\\alpha$-approximate portfolio: its size grows like $\\ln \\kappa / \\ln(1/\\alpha)$, where $\\kappa$ is the ratio between the largest and smallest rewards, and it solves the welfare-maximization problem only $\\tilde{O}((\\ln \\kappa)^2 \\ln \\ln N / \\ln(1/\\alpha))$ times. If correct, this means decision-makers can inspect a handful of concrete policies, each tied to a distinct fairness stance, instead of committing to a single welfare function before seeing the consequences. Experiments on a taxi gridworld, a post-disaster allocation problem, and a healthcare intervention task find portfolios of at most ten policies with approximation factors near 1.","feed_headline":"A few policies can cover every fairness preference","feed_subtitle":"New algorithm builds a small portfolio that stays near-optimal for every p-mean welfare function.","key_machinery":"Two objects carry the argument: the generalized $p$-mean $f(x, p) = (1/N) (\\sum_i x_i^p)^{1/p}$, with limiting interpretations $f(x, -\\infty) = \\min_i x_i$ and $f(x, 0) = (\\prod_i x_i)^{1/N}$, and the two aggregation rules $v^{(1)}(\\pi, p) = \\mathbb{E}_{\\tau}[f(G(\\tau), p)]$ and $v^{(2)}(\\pi, p) = f(\\mathbb{E}_{\\tau}[G(\\tau)], p)$. The engine is a line search that, starting from a policy optimal at $p$, bisects $[p, 1]$ to find the largest $b$ such that the same policy is $\\alpha$-approximate on the whole interval $[p, b]$; monotonicity of $p$-means in $p$ and a Lipschitz bound $d \\ln v^{(\\ell)}_*(p)/dp \\leq \\kappa \\ln \\kappa$ on the logarithm of the optimal value function convert interval shrinking into the stated size and oracle bounds.","core_discovery":"The paper's central claim is that the entire continuum of $p$-mean welfare objectives in a multi-objective MDP can be compressed into a compact portfolio of policies with a multiplicative guarantee. Formally, Theorem 4.1 states that for an MDP with $N$ reward functions bounded by $L$ and $U$, condition number $\\kappa = U/L$, feasible policy set $\\Pi$, aggregation rule $\\ell \\in \\{1, 2\\}$, and $\\alpha \\in (0, 1)$, p-MEAN PORTFOLIO returns a portfolio $\\Pi'$ such that for every $p \\leq 1$ there exists $\\pi \\in \\Pi'$ with $v^{(\\ell)}(\\pi, p) \\geq \\alpha \\cdot \\max_{\\pi' \\in \\Pi} v^{(\\ell)}(\\pi', p)$. The portfolio size is $O(\\ln \\kappa / \\ln(1/\\alpha))$ and the number of oracle calls to the welfare-maximization subroutine is $\\tilde{O}((\\ln \\kappa)^2 \\ln \\ln N / \\ln(1/\\alpha))$, with logarithmic factors suppressed. The construction starts from $p_0 = -\\ln N / \\ln(1/\\alpha)$, whose optimal policy is already $\\alpha$-optimal for the entire tail $p \\leq p_0$, then advances through increasing $p$ values chosen by a bisection line search, ending at $p = 1$.","pith_inferences":["The portfolio's members are not arbitrary policies but optima for distinct $p$ values, so the algorithm implicitly produces a navigable map of the fairness frontier; the paper illustrates this but does not develop it as an interface.","In multi-stakeholder RLHF-style alignment, the same construction could offer human stakeholders a short menu of alignment policies corresponding to different welfare aggregations instead of requiring one aggregation rule to be fixed in advance.","The condition-number dependence suggests a stress test the paper does not run: when stakeholder reward scales are highly unequal, the guaranteed portfolio grows, and equalizing reward scales before portfolio construction would shrink it.","Since the theorem assumes an exact oracle, an immediate extension is to quantify the oracle gap empirically in small MDPs where exact optima are enumerable, turning the heuristic comparisons into measured approximation guarantees."],"forward_implications":["A decision-maker no longer needs to commit to a single $p$ in advance: a portfolio whose size grows logarithmically in the reward ratio contains, for every $p \\leq 1$, a policy within factor $\\alpha$ of the $p$-mean optimum.","The guarantee holds for both ESR and SER aggregation, so the method covers policies executed once per trajectory and policies whose rewards accumulate over repeated executions.","The oracle-cost bound means that improving $\\alpha$ by a constant factor changes the number of welfare-maximization problems solved by only a constant factor, making higher-fidelity portfolios cheap in the oracle model.","Because nearby $p$ values have nearby optimal policies (Proposition 4.2), solving the next welfare-maximization problem can be warm-started from the previous solution, with a gap bounded by $(q - p) U H \\kappa \\ln \\kappa$.","The budget-constrained heuristic gives a practical fallback when exact oracle calls are too expensive; in the reported experiments it matches the full algorithm's coverage with far fewer calls."],"supporting_citations":[{"why":"Supplies the monotonicity of generalized p-means (Lemma 3.2), the property that lets the line search extend approximation intervals.","marker":"Bullen, 2003"},{"why":"Defines p-mean welfare objectives for MORL and provides Welfare Q-Learning, the oracle used in the taxi experiments.","marker":"Fan et al., 2023"},{"why":"Develops algorithms for non-linear scalarized objectives, the starting point for the v(2)/SER aggregation rule.","marker":"Agarwal et al., 2022"},{"why":"Provides the Social Choice Language Model that turns natural-language preferences into the 59 reward functions and 200 feasible policies in the healthcare experiment.","marker":"Verma et al., 2024a"},{"why":"Supplies the index heuristic used to approximately solve the restless bandit MDP in the healthcare domain.","marker":"Whittle, 1988"},{"why":"Gives portfolio-size trade-offs for all-p fair facility location, the combinatorial analogue whose bounds the RL result extends.","marker":"Gupta et al., 2023; 2025"},{"why":"Provides the axiomatic characterization of generalized p-means used to justify studying this family of welfare functions.","marker":"Roberts, 1980"},{"why":"Establishes that optimally controlling an RMAB is intractable, motivating the approximate Whittle-index solver behind the healthcare portfolios.","marker":"Papadimitriou and Tsitsiklis, 1999"}],"fun_headline_variants":["A tiny portfolio of policies covers every p-mean welfare goal","One small portfolio, near-optimal for every social welfare preference","Compress all fairness objectives into a few RL policies","Small policy set, near-optimal for every p-mean welfare","A compact policy portfolio that satisfies all p-mean social welfare"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an oracle can solve the welfare-maximization problem $\\max_{\\pi \\in \\Pi} v^{(\\ell)}(\\pi, p)$ exactly for every $p$ it is asked about, and that every reward is strictly positive so the condition number $\\kappa = U/L$ is defined; if either fails, Theorem 4.1's size and oracle bounds do not govern what the algorithm actually delivers.","fun_headline_variants_meta":{"raw":{"variants":["A tiny portfolio of policies covers every p-mean welfare goal","One small portfolio, near-optimal for every social welfare preference","Compress all fairness objectives into a few RL policies","Small policy set, near-optimal for every p-mean welfare","A compact policy portfolio that satisfies all p-mean social welfare"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2970,"prompt_tokens":1046,"completion_tokens":1924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1839}},"tokens_in":662,"tokens_out":1924,"duration_ms":15328,"temperature":1.0,"reasoning_tokens":1839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:42:42.157166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run p-MEAN PORTFOLIO on a small MDP with $N = 2$ rewards where exact optimal policies are enumerable, with fixed $L$, $U$ and $\\alpha = 0.5$, and test every $p$ in a fine grid: finding any $p \\leq 1$ where no portfolio policy reaches $\\alpha$ times the true optimum, or any instance where the returned portfolio exceeds the Theorem 4.1 size bound, would refute the theorem. A second test is to allow one reward to be zero, which makes $\\kappa$ undefined and shows whether the strictly-positive assumption is essential.","supporting_citations":[],"review_version":1}