{"id":"669e656a-5611-4c19-83ea-0e899cc6c1ea","arxiv_id":"2507.14066","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Pb-MORL learns a multi-objective reward model from preference comparisons and claims to achieve Pareto-optimal policies, outperforming an oracle in energy and highway tasks.","lead":"This paper combines preference-based reward learning with multi-objective reinforcement learning, letting a teacher's pairwise comparisons of trajectory segments train a multi-objective reward model that guides a single weight-conditioned policy. If preferences are reliable, this could remove the need to hand-design reward functions for multi-objective control tasks such as energy management and autonomous driving.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 4 fails for Pareto frontiers with supported interior points: the supporting weight set is a singleton, so no finite weight grid can recover such policies, and Theorem 2's entire-frontier guarantee is false.","rationale":"The reader's weakest assumption identifies Assumption 4 as the key failure point, and my analysis confirms this is the most load-bearing concern. The paper's central theoretical contribution is the claim that preferences can derive policies across the entire Pareto frontier, presented in Theorems 1-3 and supported by Algorithm 1. That claim rests directly on Assumption 4, which is used to justify replacing the continuous weight space with a finite grid. The assumption is not merely hard to satisfy; it is false in simple finite MDPs with linear scalarization. A supported Pareto-optimal policy lying in the relative interior of a linear segment of the Pareto front is optimal only along a measure-zero weight set, so no epsilon-ball around that weight preserves optimality of any single policy. This is a concrete, checkable counterexample rather than a matter of interpretation. The empirical results, while suggestive, do not compensate for the theoretical collapse: the figures show mean curves over five seeds without error bars or significance tests, and the 'surpasses the oracle' claim on energy and highway tasks is presented without confidence intervals. The paper does have merits: the Pb-MORL framework is clearly described, the integration of preference learning with Envelope Q-Learning is straightforward, and the experiments cover multiple domains. However, the central guarantee of full Pareto-frontier recovery fails under Assumption 4, and Theorem 4's perfect-alignment premise is a tautology that does not transfer to a learned reward model. Since the reader already rejected the paper on these grounds, my independent stress test does not change the verdict.","tokens_in":20445,"tokens_out":11825,"duration_ms":552158,"concrete_test":"Instantiate a finite two-objective MDP with three deterministic policies having returns A=(0,1), B=(1,0), C=(0.5,0.5), and verify that C is Pareto-optimal. Run Algorithm 1 for an epsilon > 0 and a finite grid W[Nw] chosen so that the singleton weight (0.5,0.5) is not a grid point, using the true reward function r. Theorem 2 predicts that C must be returned; because C is optimal only at that single weight, no grid point will yield C as the unique weighted-return maximizer, so C will be absent from Pi*. If C is absent, the claim that Algorithm 1 obtains the entire convex Pareto frontier is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 4 (Section III.A) states that for every w0 there is an epsilon-ball on which the w0-optimal policy remains optimal. This is false even in a finite two-objective MDP. Let the policy set contain three deterministic policies with returns A=(0,1), B=(1,0), C=(0.5,0.5). C is Pareto-optimal and supported, but the set of weights where C maximizes w^T R is exactly the singleton {w : w1 = w2 = 0.5}. At that w0 the argmax set is {A,B,C}; for any epsilon > 0, nearby weights make A or B strictly better, so no single policy from the argmax set remains optimal throughout the ball. Hence the discretization W[Nw] with cells of side epsilon cannot represent C, and Algorithm 1 will never return C for a generic grid, contradicting Theorem 2's claim that Algorithm 1 obtains the entire convex Pareto frontier. The same failure affects non-convex frontiers whenever a Pareto-optimal policy is supported only on an indifference boundary. Theorem 3 does not rescue the claim: Algorithm 2/4 requires enumerating the full policy space Pi and uses strict inequalities w_k^T R(pi_i) > w_k^T R(pi_j) for every unit weight, which misclassifies Pareto dominance when the two policies tie on one objective (e.g., R'=(1,2) is dominated by R*=(2,2), but no unit weight gives a strict inequality in all coordinates). Thus the central theoretical guarantee that preferences recover the entire Pareto frontier is not established, even with a perfect teacher and ground-truth rewards.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Pb-MORL, a framework that replaces hand-designed multi-objective rewards with an explicit multi-objective reward model trained from preference feedback and then optimizes a weight-conditioned policy via Envelope Q-Learning. The authors make four theoretical claims: optimal policies under any weight are Pareto optimal (Theorem 1); traversing a finite weight grid recovers the entire convex Pareto frontier (Theorem 2); non-convex frontiers can be recovered through unit-weight comparisons (Theorem 3); and optimizing a reward model that is perfectly aligned with teacher preferences is equivalent to selecting the teacher-optimal policy (Theorem 4). The experimental section compares Pb-MORL against EQL with ground-truth rewards on three benchmarks, an energy-storage task, and a highway-driving task, and claims that Pb-MORL surpasses the oracle on the latter two domains.","tokens_in":20862,"tokens_out":9469,"duration_ms":96665,"significance":"The motivation is strong: preference-based learning can reduce reward-engineering burden in multi-objective control, and combining PbRL with MORL is a timely problem. The proposed pipeline is simple and the scripted-teacher evaluation is reproducible in spirit. However, the theoretical guarantees are the paper's central contribution and they are not established. Theorem 4 is circular because its perfect-alignment premise already contains the conclusion. Assumption 4, which is needed for the finite-grid frontier claim, is false in generic finite MDPs with ties. Theorem 3's unit-weight dominance test misclassifies weak dominance. The empirical claim of surpassing the oracle also lacks statistical support. If the theorems were correct, the paper would be a significant step for preference-based MORL; as submitted, the headline claims are unsupported.","major_comments":[{"comment":"The statement to be proved is already contained in the perfect-alignment assumption. Equation (15) asserts that for every segment pair and every length H, the teacher's preference ordering coincides with the ordering by discounted w^T r-hat; letting H become arbitrarily long gives Eq. (16), which is exactly the policy-level equivalence claimed by the theorem. The proof then assumes a policy pi' with larger r-hat return, invokes Eq. (15) to infer that the teacher would prefer pi', and writes Eq. (19) as a contradiction. This rewrites the premise rather than proving that optimizing a learned reward model selects the teacher-optimal policy. No argument shows that a reward model trained by minimizing the cross-entropy loss (13) satisfies (15) even approximately. The displayed implication in Eq. (19) also appears to have the preference direction reversed: if pi' has the larger r-hat return, perfect alignment would give pi' ≻_w pi*, which is the assumed inequality, not a contradiction.","section":"Section III.B, Theorem 4 (Eqs. (15)-(19))"},{"comment":"Assumption 4 requires the argmax policy to be constant on an epsilon-ball around every weight w0. This fails whenever the optimal policy changes at a tie. For example, consider three deterministic policies with returns A=(0,1), B=(1,0), and C=(0.5,0.5). Policy C is Pareto-optimal, but it is the unique maximizer of w^T R only at w=(0.5,0.5); for every epsilon>0 there are weights in the ball where A or B is strictly better. Hence no finite grid W[N_w] with cells of side epsilon can contain a weight at which C is the unique optimal policy, and Algorithm 1 will never return C for a generic grid. This contradicts Theorem 2's claim that Algorithm 1 obtains the entire convex Pareto frontier, and it also weakens Theorem 1, whose proof relies on comparing policies at a weight in the grid. The authors would need an epsilon chosen per policy, based on the gap between value functions, but no such construction is given.","section":"Section III.A, Assumption 4 and Theorem 2"},{"comment":"The dominance test uses strict inequalities w_k^T R(sigma_i) > w_k^T R(sigma_j) for all unit weights w_k in W_I. This misclassifies Pareto dominance whenever the dominating policy ties the dominated policy on at least one objective. For instance, R'=(1,2) is dominated by R*=(2,2), but w_1^T R'=1 < 2 and w_2^T R'=2 = 2, so no unit weight yields strict inequality in every coordinate. Algorithm 2 would not remove R', and Algorithm 4 would keep both policies in the estimated Pareto set. Thus Theorem 3's claim that an arbitrary Pareto frontier can be completely obtained via unit-weight comparisons is false as stated.","section":"Section III.A, Theorem 3 and Algorithms 2 and 4"},{"comment":"The abstract and conclusion claim that Pb-MORL 'surpasses' the oracle method on the energy and highway tasks, but the figures plot only means over five seeds, without error bars, confidence intervals, or significance tests. Because the scripted teacher is constructed from the same ground-truth reward that EQL uses directly, the claim that a learned proxy outperforms the reward-based oracle is surprising and needs statistical support. The post-hoc explanations in the text (e.g., preference emphasis on relative benefits) are not evidence. This is a load-bearing empirical claim and should be quantified or softened.","section":"Section IV.C-D, Figs. 4 and 5"}],"minor_comments":[{"comment":"Algorithms 1 and 2 iterate 'for each pi_i in Pi' over the full policy space, which is infinite and not enumerable in general; the theorems should state explicitly that these are exhaustive-search arguments or restrict Pi to a finite policy class.","section":"Section III.A, Algorithms 1-2"},{"comment":"The loss in Eq. (13) writes p(0) and p(1) with p in {0,0.5,1}; it is unclear how the indeterminate preference p=0.5 is encoded in the cross-entropy loss and whether such comparisons are used for training.","section":"Section II.B, Eq. (13)"},{"comment":"The proof of Theorem 1 uses H tending to infinity but Algorithm 1 operates with finite segments; Corollary 1 introduces an extra delta>0 lower bound on return differences that is not implied by Assumptions 1-4, so the finite-horizon guarantee depends on an unstated condition.","section":"Section III.A, Theorem 1 proof and Corollary 1"},{"comment":"The formula for the grid size, N_w = |W|/epsilon^m <= epsilon^{-m}, ignores the normalization of the simplex and should be stated as (1/epsilon)^{m-1} or similar; the expression as written is dimensionally inconsistent.","section":"Section III.A, Assumption 4"}],"recommendation":"reject","confidential_remarks":"The mathematical flaws are fundamental rather than cosmetic: Theorem 4's proof is circular, Assumption 4 is false in generic finite MDPs and is needed for the frontier-coverage claims, and Theorem 3's dominance test fails on ties. These cannot be repaired by local edits without substantially changing the paper's claims. I see no indication of deliberate misconduct; the contribution would benefit from being reframed with correct, weaker guarantees and honest empirical uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this one is a mixed bag. The Pb-MORL algorithm is what you'd expect from the title: a preference-based reward model (Bradley-Terry loss) feeding Envelope Q-learning, with a weight-conditioned policy. That's a legitimate integration, and the energy and highway environments give the empirical part some real-world texture. If the experimental results hold up, the approach has practical value.\n\nThe theory, unfortunately, does not hold up. Theorem 4 is circular in effect: it assumes the learned reward model perfectly matches the teacher's preferences (Eq. 15) and then 'proves' that optimizing that model selects the teacher's optimal policy. That is the assumption, not a derivation. Theorems 1-2 rely on Assumption 4, which says the w0-optimal policy stays optimal in an epsilon-ball around w0. This is simply false for Pareto-optimal policies supported only at a single weight; the stress-test example with A=(0,1), B=(1,0), C=(0.5,0.5) is exactly right. So the claim of recovering the entire Pareto frontier is not established. Theorem 3's Algorithm 4 also misclassifies a dominated policy that ties an objective with a Pareto-dominant one, because it uses strict inequalities per unit weight. The paper even contains a limitation paragraph acknowledging the assumptions may not hold in practice, which is honest but doesn't rescue the theorems.\n\nThe empirical section is the weakest link. Figures are training curves without standard deviation bands or significance tests, and the headline claim of 'surpassing the oracle' is not supported by any statistical evidence. In the benchmark tasks, the results look comparable, and the FT hypervolume advantage is plausible but underreported.\n\nThat said, the paper is clearly not junk. The algorithm is sensible, the writing is readable, and the real-world tasks are a step up from pure tabular benchmarks. With the theory weakened to a consistency statement and the experiments properly quantified, it could be a reasonable contribution. I'd send it to peer review rather than desk-reject: reviewers can push the authors to fix the Assumption 4 gap and add confidence intervals. A desk reject would lose a potentially useful empirical study.","headline":"Sensible PbRL+MORL integration, but the theory is tautological or rests on a false assumption, and the empirical claims lack statistics.","tokens_in":21342,"tokens_out":2995,"would_cite":false,"duration_ms":36102,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise preferences between trajectory segments can replace hand-designed reward functions and still recover all Pareto-optimal policies in multi-objective reinforcement learning.","keywords":["preference-based reinforcement learning","multi-objective reinforcement learning","Pareto frontier","reward modeling","Bradley-Terry model","Envelope Q-learning","weight-conditioned policy","autonomous driving"],"falsifier":"In a two-objective MDP with a known ground-truth reward, train the scripted teacher and reward model $\\hat{r}$ as in the paper, then search for a policy that maximizes $\\sum_t \\gamma^t w^\\top \\hat{r}$ but is dominated under the true reward for the same $w$; if such a policy exists, Theorem 4's equivalence fails. Alternatively, construct an MDP whose Pareto frontier has a single sharp corner at an isolated weight $w_0$ and run Algorithm 1 on a grid not containing $w_0$: if the corner policy is not returned, the claim that the entire frontier is covered by the finite grid fails.","tokens_in":20244,"feed_emoji":"🎯","tokens_out":8025,"duration_ms":82994,"temperature":0.7,"pith_summary":"This paper claims that pairwise preference judgments, provided by a teacher as 'segment A is better than segment B under weight w', carry enough information to train multi-objective reinforcement learning policies across the entire Pareto frontier, without hand-designing a vector reward. It proves that a consistent teacher's preferences recover the convex Pareto frontier by traversing weights, and the non-convex parts of the frontier by comparing policies against unit-vector weights. The load-bearing construction is a multi-objective reward model trained with a Bradley-Terry preference model; Theorem 4 shows that if this reward model perfectly matches the teacher, maximizing the weighted discounted return is equivalent to selecting the optimal policy for that weight. On benchmark, multi-energy management, and multi-lane highway tasks, the method performs on par with or better than an oracle that trains on the ground-truth reward function.","feed_headline":"Preferences pick out the whole Pareto frontier without reward design","feed_subtitle":"Pairwise 'prefer A over B' judgments train a reward model that beats ground-truth rewards on energy and driving tasks.","key_machinery":"The machinery is a learned multi-objective reward model $\\hat{r}_\\psi$: a vector-valued reward in $\\mathbb{R}^m$ trained by minimizing the cross-entropy loss (13) under a Bradley-Terry preference predictor (14), so that $\\exp\\big(\\sum_t \\gamma^t w^\\top \\hat{r}(s_t^1,a_t^1)\\big)\\big/\\sum_i \\exp\\big(\\sum_t \\gamma^t w^\\top \\hat{r}(s_t^i,a_t^i)\\big)$ predicts the teacher's probability of preferring segment 1 under weight $w$. This reward model replaces the hand-designed scalar reward and is consumed by Envelope Q-learning, whose Bellman optimality operator with an optimality filter over $(a,w)$ guarantees convergence of the multi-objective Q-function. Coverage of the whole frontier rests on Assumption 4, which makes the optimal policy locally constant in $w$ and reduces the weight space to a finite grid $W[N_w]$, plus the identity-weight comparison procedure of Algorithm 2 (Theorem 3) for non-convex frontiers.","core_discovery":"The central discovery is a preference-based route to Pareto-optimal policies in multi-objective RL. Given a teacher who can compare trajectory segments under any weight vector, the paper constructs an explicit multi-objective reward model $\\hat{r}_\\psi$ using the Bradley-Terry model and cross-entropy loss (Eqs. 13–14), then feeds that reward model into Envelope Q-learning to obtain a policy $\\pi_\\phi(a|s,w)$. The paper proves Theorem 4: if $\\hat{r}$ satisfies Eq. (15), i.e., it ranks any two segments exactly as the teacher does under any weight, then maximizing $J(\\pi)=\\sum_t \\gamma^t w^\\top \\hat{r}(s_t,a_t)$ is equivalent to picking the optimal policy $\\pi^*(\\cdot|\\cdot,w)$. Theorems 1–2 show that the weight-traversal Algorithm 1 yields exactly the convex Pareto frontier as the segment length $H\\to\\infty$, and Theorem 3 with Algorithm 2 shows that unit-vector weight comparisons recover non-convex frontiers. Empirically, Pb-MORL matches the oracle on Deep Sea Treasure, Fruit Tree, and Resource Gathering, and surpasses it on the energy and highway tasks.","pith_inferences":["If reward-model error is bounded, Theorem 4 likely degrades gracefully: the suboptimality gap between the learned and true Pareto policy should scale with the worst-case misranking probability, a bound the paper does not derive but could be tested on the benchmark tasks.","The theorem's perfect-alignment condition (Eq. 15) is unachievable with finite preference data; in practice the method's success depends on the scripted teacher's preferences matching the ground-truth reward, so real-human evaluation would be the decisive test of whether the preference signal alone is sufficient.","Assumption 4's local-constancy is the fragile step in the frontier-coverage claim: the finite grid $W[N_w]$ can miss a policy that is optimal at an isolated weight, so the empirical near-oracle results do not by themselves establish the theoretical claim of whole-frontier coverage."],"forward_implications":["Replacing hand-designed multi-objective rewards with pairwise preference data turns any convergent MORL algorithm (the paper uses Envelope Q-learning) into a Pareto-optimal policy learner, per Theorem 4.","The full convex Pareto frontier is reachable by sweeping weights (Theorem 2), and non-convex frontiers are reachable by comparing policies under unit-vector weights (Theorem 3), so no special treatment is needed for non-convex trade-offs.","The learned policy $\\pi_\\phi(a|s,w)$ is weight-conditioned, so changing the objective trade-off at deployment only requires changing $w$, not retraining; the paper demonstrates this in energy management and highway driving.","On the energy and highway tasks, preference-driven learning outperformed the oracle that trains on ground-truth rewards, suggesting preferences act as a stabilizer against overfitting to immediate objectives."],"supporting_citations":[{"why":"Supplies the trajectory-segment preference framework and the Bradley-Terry reward-learning loss that Pb-MORL extends to multiple objectives.","marker":"[11]"},{"why":"Provides Envelope Q-learning, the MORL algorithm whose convergence Pb-MORL relies on to optimize the learned reward model.","marker":"[33]"},{"why":"The Bradley-Terry paired-comparison model used in Eq. (14) to turn reward sums into preference probabilities.","marker":"[43]"},{"why":"Source of the preference assumptions (symmetry, consistency, transitivity) and the relabeling technique; also the scripted-teacher construction.","marker":"[9]"},{"why":"Informs the continuous preference collection and data-efficiency techniques used in Algorithm 3.","marker":"[27]"},{"why":"Supplies the benchmark tasks (DST, FT, RG) and the Expected Utility and Hypervolume evaluation metrics.","marker":"[30]"},{"why":"Motivates the query-policy-aligned replay buffer so the reward model stays accurate in the near-policy region.","marker":"[47]"},{"why":"Defines the Deep Sea Treasure environment used as a benchmark.","marker":"[48]"},{"why":"Defines the Resource Gathering environment used as a benchmark.","marker":"[49]"},{"why":"Provides the multi-lane highway environment used for the autonomous driving experiment.","marker":"[50]"}],"fun_headline_variants":["Preference feedback reaches full Pareto frontier in multi-objective RL","Pairwise preferences beat handcrafted rewards across Pareto frontier","No reward design: preferences derive entire Pareto-optimal policy set","Preferences train reward model that beats oracle on real tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole proof chain hinges on the reward model $\\hat{r}$ perfectly reproducing the teacher's preferences for every pair of segments and every weight (Eq. 15), and on the optimal policy being constant within an $\\epsilon$-ball of every sampled weight (Assumption 4); if either fails, a learned or finite-grid policy can fall outside the true Pareto frontier.","fun_headline_variants_meta":{"raw":{"variants":["Preference feedback reaches full Pareto frontier in multi-objective RL","Pairwise preferences beat handcrafted rewards across Pareto frontier","No reward design: preferences derive entire Pareto-optimal policy set","Preferences train reward model that beats oracle on real tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3084,"prompt_tokens":976,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2040}},"tokens_in":592,"tokens_out":2108,"duration_ms":16570,"temperature":1.0,"reasoning_tokens":2040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:01:58.272332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a two-objective MDP with a known ground-truth reward, train the scripted teacher and reward model $\\hat{r}$ as in the paper, then search for a policy that maximizes $\\sum_t \\gamma^t w^\\top \\hat{r}$ but is dominated under the true reward for the same $w$; if such a policy exists, Theorem 4's equivalence fails. Alternatively, construct an MDP whose Pareto frontier has a single sharp corner at an isolated weight $w_0$ and run Algorithm 1 on a grid not containing $w_0$: if the corner policy is not returned, the claim that the entire frontier is covered by the finite grid fails.","supporting_citations":[{"cited_title":"Learning all optimal policies with multiple criteria,","cited_arxiv_id":null,"evidence_quote":"Defines the Resource Gathering environment used as a benchmark."},{"cited_title":"Query-policy mis- alignment in preference-based reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Motivates the query-policy-aligned replay buffer so the reward model stays accurate in the near-policy region."},{"cited_title":"A generalized algorithm for multi-objective reinforcement learning and policy adaptation,","cited_arxiv_id":null,"evidence_quote":"Provides Envelope Q-learning, the MORL algorithm whose convergence Pb-MORL relies on to optimize the learned reward model."},{"cited_title":"Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsuper- vised pre-training,","cited_arxiv_id":null,"evidence_quote":"Source of the preference assumptions (symmetry, consistency, transitivity) and the relabeling technique; also the scripted-teacher construction."},{"cited_title":"Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Informs the continuous preference collection and data-efficiency techniques used in Algorithm 3."},{"cited_title":"A toolkit for reliable benchmarking and research in multi-objective reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark tasks (DST, FT, RG) and the Expected Utility and Hypervolume evaluation metrics."},{"cited_title":"Empirical evaluation methods for multiobjective reinforcement learning algorithms,","cited_arxiv_id":null,"evidence_quote":"Defines the Deep Sea Treasure environment used as a benchmark."}],"review_version":1}