Pith. sign in

REVIEW 2 major objections 6 minor 21 references

Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

T0 review · 2 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When experts label trajectories incomparable, a multi-objective Bradley-Terry model recovers multi-dimensional rewards and the Pareto frontier of policies.

desk verdict Clean theoretical bridge from PbRL to multi-objective settings via an explicit incomparability model; the math holds, the synthetic experiments mainly check self-consistency. read the letter →

arxiv 2607.11432 v1 pith:M5WGF5EU submitted 2026-07-13 cs.LG

Simone Drago , Marco Mussi , Leonardo Bianconi , Alberto Maria Metelli This is my paper · ORCID
classification cs.LG
keywords preference-basedreinforcementlearningmulti-objectiveRLBradley-TerrymodelincomparabilityParetofrontierhumanfeedbackrationalitymodelscomparison-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Preference-based reinforcement learning has treated an expert’s refusal to rank two trajectories as noise or indifference. This paper argues the refusal is often rational: it signals that multiple conflicting objectives make neither trajectory dominate the other. The authors formalize Markov decision processes with four-way comparisons (prefer, prefer-inverse, indifferent, incomparable), list four mode-desiderata any rationality model must satisfy, and introduce a multi-objective Bradley-Terry model whose scores use the average utility difference for clear preferences and the standard deviation of the difference vector for incomparability. They prove that the non-convex negative log-likelihood still admits sample-complexity guarantees on the KL divergence between true and estimated comparison distributions, and they show in simulation that the recovered multi-dimensional reward produces policies whose Pareto front approaches the true front. A sympathetic reader cares because the approach turns an everyday human response into a signal that lets multi-objective reinforcement learning run on ordinary pairwise feedback instead of dense vector rewards.

What carries the argument

The multi-objective Bradley-Terry model (MOBT): a four-class softmax whose incomparability score is defined as √d times the standard deviation of the utility-difference vector; this single geometric quantity makes incomparability the modal outcome precisely on the non-standard diagonals where objectives conflict.

What would settle it

Collect real human comparison labels (including free-form “cannot compare”) on a multi-objective control task whose ground-truth objectives are known, fit MOBT, and test whether the recovered reward matrices and the hypervolume of the induced Pareto front match the known objectives within the reported KL and hypervolume ratios.

Watch

Extended reading notes

Core claim

The multi-objective Bradley-Terry (MOBT) model—softmax of four scores that are the signed average utility difference for direct and inverse preference, a constant for indifference, and the standard deviation of the utility-difference vector plus bias for incomparability—satisfies the four natural mode conditions of Definition 2.2, falls back to ordinary Bradley-Terry when the problem is one-dimensional, and can be learned from an offline dataset so that the induced comparison distribution is close in KL to the expert’s.

Load-bearing premise

Every experimental label is generated by a synthetic expert that itself follows the proposed MOBT model, so reconstruction success mainly verifies recovery of parameters from the same family that produced the data.

Editorial extensions

If this is right

  • Offline datasets that retain “incomparable” labels can reconstruct multi-dimensional rewards without ever receiving dense multi-objective reward vectors.
  • Standard single-objective preference models that discard or re-label incomparabilities recover only one scalarization and cannot traverse the Pareto front.
  • KL error between true and estimated comparison distributions scales as O(R Λ d k √(log(1/δ)/N)) under linear utility features.
  • Even a local minimum of the non-convex likelihood still yields KL error controlled by the marginal probability of incomparability.
  • The recovered multi-dimensional reward can be handed to any multi-objective RL solver to obtain the Pareto set of policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Allowing an explicit “cannot compare” answer may lower cognitive load relative to forcing experts to state scalarization weights or multi-criteria scores.
  • High estimated incomparability bias or frequent incomparability labels could serve as a diagnostic that a preference dataset is multi-objective rather than noisy.
  • An online active-learning variant that chooses trajectory pairs expected to reduce incomparability uncertainty would be a direct algorithmic extension.
  • The same score construction could be ported to ranking or social-choice settings where partial orders arise from conflicting criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper formalizes comparison-based RL (CbRL) via the Markov Decision Process with Comparisons (MDPC), allowing four outcomes (direct preference, inverse preference, indifference, incomparability). It states four mode-desiderata for a rationality model (Definition 2.2), proves that no model satisfying them can yield a convex negative log-likelihood (Proposition 2.1), and introduces the multi-objective Bradley-Terry (MOBT) model with scores (7)–(10). Lemma 3.1 shows compliance under α>0, α>β; Theorems 3.2–3.3 give KL sample-complexity bounds of order RΛdk √(log(1/δ)/N) for the global optimum and an additive P(∥) term for local optima under linear features and boundedness. Experiments on synthetic GridWorld, MO-Hopper and LQR data (labels drawn from MOBT itself) report low test KL, multi-dimensional reward recovery, and LQR hypervolume ratios approaching 0.96 at 5k pairs, plus a robustness sweep over mistake probability ε.

Significance. If the modeling assumptions hold, the work cleanly bridges PbRL and MORL by treating incomparability as a rational signal rather than noise, and supplies the first sample-complexity guarantees for a multi-dimensional rationality model under offline comparisons. The impossibility of convex NLL (Prop. 2.1), the explicit desiderata, the reduction to classical BT when d=1 (Remark 3.1), and the local-optimum bound that isolates the incomparability probability are technically solid contributions. The LQR closed-form Pareto evaluation and the total-variation comparison against BT/RK/Davidson baselines (Appendix D.4) strengthen the empirical case inside the model class. The main limitation is that all labels are generated by the same MOBT family that is recovered, so external validity of the standard-deviation incomparability score remains untested; the theoretical results themselves do not depend on that loop.

major comments (2)
  1. Section 4 and Appendix D.1: every experimental label is drawn from the MOBT softmax of Eqs. (7)–(10) itself (“we have modeled our synthetic expert to use our MOBT as its rationality model”). Consequently the reported KL values, reward matrices (Fig. 3) and LQR hypervolume ratios (0.86–0.96) demonstrate self-consistency of the optimizer rather than that real human incomparability obeys h_∥=√d·std(δ)+β. The central claim that the model “recovers the Pareto frontier of policies” from comparison feedback therefore rests on an untested inductive bias. At minimum the paper should (i) state this limitation prominently in the abstract and Section 4, and (ii) either supply a non-MOBT synthetic expert (e.g., Thurstone-style multi-objective noise or a lexicographic rule) or a small human pilot that records genuine incomparability labels.
  2. Theorem 3.3 / Eq. (13): the additive error term is proportional to the marginal probability of incomparability P_θ*(∥). In the multi-objective regimes the paper targets this probability is expected to be non-negligible (and is the source of non-convexity). The bound therefore does not guarantee that a local optimum recovered by ADAM is close to the expert’s distribution when conflict is high. The manuscript should either (a) quantify how large P(∥) can be under the desiderata before the additive term dominates, or (b) provide empirical evidence that the local optima found in practice remain useful for Pareto recovery even when the incomparability ratio reaches the 0.4–0.6 range examined in Table 4 of the appendix.
minor comments (6)
  1. Definition 2.2, Eq. (4): the limit is written “lim_δ→+∞ t” with t∈{−1,1}^d∖{1_d,−1_d}; a short clarifying sentence that the limit is taken along the ray c·t, c→+∞, would remove ambiguity.
  2. Figure 1 caption and surrounding text: the 2-D illustration is helpful but the axes are labeled only δ1, δ2; adding the four mode regions explicitly in the figure legend would improve readability.
  3. Assumption 3.1: the feature map φ is assumed known. A brief remark on how one would estimate or over-estimate d and the feature dimension in practice (the paper already notes that d can be overestimated) would help practitioners.
  4. Table 2: report the corresponding train/test split sizes and the number of random seeds more prominently; the 95 % C.I. notation is clear but the absolute number of runs is easy to miss.
  5. Appendix B derivation of h_∥: the projection argument is correct, yet the final step equates the Euclidean distance to √d·std; a one-line identity ||x−x̄1||_2 = √d·std(x) would make the algebra self-contained.
  6. Typographical: “thereinforcement learning” (Abstract), “asincomparable” (Abstract), and occasional missing spaces after commas in the arXiv text should be cleaned.

Circularity Check

1 steps flagged · score 2.0 of 10

Theory (desiderata → MOBT compliance → sample-complexity) is self-contained and non-circular; only the simulated-expert experiments are mildly self-referential.

  1. other [Section 4 / Appendix D.1]
    "we have modeled our synthetic expert to use our MOBT as its rationality model. When shown a pair of trajectories, we provide the expert also with the step-wise reward vectors for each trajectory, which the expert then employs to compute the score functions according to Equations (7) to (10), and finally generating a label by sampling from the distribution defined as the soft-max of the scores."

    Every comparison label used for training and for the reported KL / hypervolume numbers is sampled from the same four-score softmax that the optimizer later recovers. Success therefore demonstrates that ADAM can invert the generative process that produced the data, not that the standard-deviation incomparability score matches external human judgments. The theoretical guarantees remain unaffected.

full rationale

Definition 2.2 states four independent mode-desiderata on the limits of f(·|δ). The MOBT scores (Eqs. 7–10) are then proposed and Lemma 3.1 verifies compliance by direct asymptotic comparison of the four scores; the proof never assumes the conclusion. Proposition 2.1 (impossibility of a convex NLL) and Theorems 3.2–3.3 (KL bounds via covering + Lipschitz + sub-Gaussian arguments) are standard concentration arguments that do not rely on self-citation or on the experimental data. The only mild circularity is experimental: all labels are drawn from the identical MOBT generative family that is later recovered (Appendix D.1). This verifies optimizer consistency inside the model class but does not test whether real human incomparability obeys the standard-deviation score. That limitation is acknowledged by the authors and does not infect the formal claims, so the overall circularity score remains low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

Central claims rest on the four mode-desiderata (Def. 2.2), linear feature representation of utility (Ass. 3.1), parameter and feature boundedness (Ass. 3.2), known number of objectives d, and the modeling choice that incomparability is quantified by the Euclidean distance of the utility-difference vector from the main diagonal (std). Free parameters α, β are fitted; the synthetic expert is an instance of the same model.

free parameters (3)
  • α (indifference score)
    Learnable scalar that sets the height of the indifference mode; required α > 0 and α > β for desiderata compliance.
  • β (incomparability offset)
    Learnable additive constant in the incomparability score; controls baseline probability of ∥ labels.
  • W (utility weight matrix)
    d × k matrix mapping trajectory features to multi-dimensional utilities; estimated by NLL minimization.
assumptions (4)
  • domain assumption Utility representation theorem: rational comparisons are generated from an underlying multi-dimensional utility difference via a fixed rationality model f (von Neumann–Morgenstern style).
    Invoked in Section 2 to justify writing ρ(τ ∘ τ′) = f(∘ | u(τ) − u(τ′)).
  • domain assumption Feature-based linear utility: u(τ) = W ϕ(τ) with known ϕ and unknown W (Ass. 3.1).
    Used for all sample-complexity statements; not required for the model definition itself.
  • domain assumption Boundedness of parameters and feature differences (Ass. 3.2).
    Needed for Lipschitz and sub-Gaussian arguments in Lemmas C.1–C.2 and Theorems 3.2–3.3.
  • ad hoc to paper Incomparability score equals √d · std(δ) + β (Eq. 10).
    Chosen because it grows with distance from the main diagonal; other dispersion measures could be substituted.
invented entities (2)
  • Markov Decision Process with Comparisons (MDPC)
    purpose: Formal environment that emits four-class comparison labels including incomparability.
    Direct multi-objective counterpart of the MDPP used in PbRL; new tuple (S,A,H,p,μ,ρ).
  • Multi-Objective Bradley-Terry (MOBT) model
    purpose: Four-score softmax that realizes the desiderata of Def. 2.2 and reduces to BT when d=1.
    Core technical contribution; scores defined in Eqs. 7–10.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability." pith.science (2026). https://pith.science/paper/M5WGF5EU

@misc{pith2026260711432,
  author       = {Pith},
  title        = {Pith review of: Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5WGF5EU}},
  note         = {Machine review of arXiv:2607.11432}
}
read the original abstract

In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.

Figures

Figures reproduced from arXiv: 2607.11432 by the authors.

Figure 1
Figure 1. Desiderata for a 2D CbRL problem. Definition 2.2 puts the focus of the rationality model on characterizing the mode outcome (i.e., the highest probability one) in various limits of utility differences. Intuitively, given two trajec￾tories τ, τ 1 PT , if τ dominates τ 1 , i.e., is better w.r.t. all objectives, we expect that the most prob￾able outcome that a human expert can provide is direct preference (Equation 1).… view at source ↗
Figure 2
Figure 2. GridWorld true reward matrices. and define as Pθploc its induced distribution under the same Q P ∆pT 2 q. Under Assumptions 3.1 and 3.2, for every δ P p0, 1q, it holds that: DKLpPθ˚ ∥Pθploc q ď Or ˜ RΛdkc log p1{δq N ` Pθ˚ p∥qΛd ? kR¸ , (13) where Pθ˚ p∥q :“ ş τ,τ1 Pθ˚ pdτ, dτ 1 , ∥q is the marginal probability of observing an incomparability. We observe that moving from the global optimum, which we are not guarante… view at source ↗
Figure 3
Figure 3. Estimated GridWorld reward matrices (mean over 10 runs, K=1000, M=1). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: GridWorld reward matrix estimated by baselines (mean over 10 runs, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Pareto frontier in LQR (10 runs, mean ˘ 95% C.I.). 1.0 2.0 3.0 4.0 5.0 7.5 10.0 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.225 Number of Trajectory Pairs (Thousands) KL Divergence ϵ “ 0.0 ϵ “ 0.05 ϵ “ 0.1 ϵ “ 0.25 ϵ “ 0.5 ϵ “ 0.75 ϵ “ 1.0 [PITH_FULL_IMAGE:figures/ful…
Figure 7
Figure 7. Figure 7: GridWorld reward matrix estimated using different single-objective rationality models [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 4 linked inside Pith

  1. [1]

    Concrete problems in AI safety.arXiv preprint arXiv:1606.06565,

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety.arXiv preprint arXiv:1606.06565,

    arXiv

  2. [2]

    Active preference-based gaussian process regression for reward learning.arXiv preprint arXiv:2005.02575,

    Erdem Bıyık, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning.arXiv preprint arXiv:2005.02575,

    arXiv 2005

  3. [3]

    URL https://proceedings.neurips.cc/paper_files/ paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf. Robin L. Plackett. The analysis of permutations.Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202,

    2019

  4. [4]

    Traversing pareto optimal poli- cies: Provably efficient multi-objective reinforcement learning.arXiv preprint arXiv:2407.17466,

    Shuang Qiu, Dake Zhang, Rui Yang, Boxiang Lyu, and Tong Zhang. Traversing pareto optimal poli- cies: Provably efficient multi-objective reinforcement learning.arXiv preprint arXiv:2407.17466,

    arXiv

  5. [5]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  6. [6]

    Multi-Objective Reinforcement Learning.MORL addresses the challenge of optimizing multiple, often conflicting, criteria to discover a set of Pareto-optimal policies

    13 A Extended Related Works In this appendix, we summarize the relevant literature, focusing on PbRL, rationality models alterna- tive to BT, and proposed methods for handling non-preference. Multi-Objective Reinforcement Learning.MORL addresses the challenge of optimizing multiple, often conflicting, criteria to discover a set of Pareto-optimal policies....

    2024

  7. [7]

    The CbRL setting has the same learning goal, i.e., recovering the Pareto frontier

    as an auxiliary task to enhance learning in continuous MOMDPs, providing also an algorithm that returns a weight-conditioned 8 policy that can adapt to different weights at inference time. The CbRL setting has the same learning goal, i.e., recovering the Pareto frontier. However, it operates under significantly weaker signals, namely pairwise trajectory-l...

    2017

  8. [8]

    was first formalized to integrate thelearning from preferencesand RL fields, and has since gained renewed attention with the rise of LLMs (Zhao et al., 2026). PbRL revolves around the mathematical framework of theMarkov decision process with preferences(MDPP, Wirth et al., 2017), combining an MDPzR with a probability distribution ρ to model the probabilit...

    2026

Show all 21 references
  1. [9]

    weight” to refer to the scalarization weight of the multiple objectives rather than the term “preference

    model, which models the probability 8We use the term “weight” to refer to the scalarization weight of the multiple objectives rather than the term “preference” commonly used in MORL to avoid ambiguity. 14 of observing a preference as the sigmoid of the difference in utility of...

    1975

  2. [10]

    In recent years, novel rationality models have been proposed to tackle specific limitations of BT

    in the preference generation process. In recent years, novel rationality models have been proposed to tackle specific limitations of BT. A first approach directly models the expert’s rationality, i.e., thedeterminismof their decisions, via a rationality parameter that rescales...

    2020

  3. [11]

    Finally, thegeneral preference optimization (Zhang et al.,

    tackles the problem of BT not guaranteeing a unique optimum to the MLE problem indirect preference optimization(Rafailov et al., 2023), proposing an approach based onenergy-based models(LeCun et al., 2006). Finally, thegeneral preference optimization (Zhang et al.,

    2023

  4. [12]

    addresses intransitive and/or cyclic preferences, proposing a model based on preference embeddings to capture complex structures that evade the limitations of BT’s scalar utility. Handling Non-Preference Feedback and the Role of Abstension.The standard PbRL approach prescribes...

    2017

  5. [13]

    threshold of the sensory perception of the judge

    defines a “threshold of the sensory perception of the judge”, i.e., a value such that, if the difference in utility is lower than this value, the expert is unable to state a clear preference. (Davidson, 1970), instead, proposes the probability of indifference as inversely prop...

    1970

  6. [14]

    The standard approach in the presence of incomparability in PbRL is to discard the query and consider the sample as erroneous (Christiano et al., 2017)

    formulates the inability to choose among alternatives as a rational response to a conflict between competing values. The standard approach in the presence of incomparability in PbRL is to discard the query and consider the sample as erroneous (Christiano et al., 2017). However...

    2017

  7. [15]

    With respect toα, we have: ››∇αhąpτ, τ 1|θq ›› 2 “0, ››∇αhăpτ, τ 1|θq ›› 2 “0,››∇αh—pτ, τ 1|θq ›› 2 “1, ››∇αh∥pτ, τ 1|θq ›› 2 “0

    Then, we can write the Frobenius norm of the gradient ofh ∥ as: }∇W h∥pτ, τ|θq} F ď 2? d ř i,jPJdK ˇˇpei ´e jqJWpϕpτq ´ϕpτ 1qq ˇˇ ››pei ´e jqpϕpτq ´ϕpτ 1qqJ›› Fc ř i,jPJdK ´ pei ´e jqJWpϕpτq ´ϕpτ 1qq ¯2 (28) ď 2? d d ÿ i,jPJdK }pei ´e jqpϕpτq ´ϕpτ 1qqJ}2 F (29) ď 2? d d ÿ i,jP...

    1985

  8. [16]

    By combining Equations (64) and (66) we can write: 2|c| c kpd´kq d ´ |c|p2k´dq 2d `β“ |c| ˜ 2 c kpd´kq d ´ 2k´d 2d ¸ `β(67) ě |c| ˜ 2 c d´1 d ´ d´2 2d ¸ `β(68) ě |c| ˆ? 2´ 1 2 ˙ `β(69) where Equation (68) is obtained by observing that the minimum of kpd´kq is d´1 , attained at...

    2020

  9. [17]

    First, we discuss the methodology of our experiments in Appendix D.1, covering the procedures for generating trajectories and comparison labels

    This appendix is structured as follows. First, we discuss the methodology of our experiments in Appendix D.1, covering the procedures for generating trajectories and comparison labels. Second, we report the specifications of the environment employed in our experiments in Appen...

    2015

  10. [18]

    MO-Hopper.We use the mo-hopper-2obj-v5 environment from the MO-Gymnasium suite (Fel- ten et al., 2023), the multi-objective extension of Hopper-v5 (Towers et al., 2025)

    The feature map counts state visitation frequencies: ϕpτq PR 25, where the i-th entry is the number of time steps the agent occupies celli. MO-Hopper.We use the mo-hopper-2obj-v5 environment from the MO-Gymnasium suite (Fel- ten et al., 2023), the multi-objective extension of ...

    2023

  11. [19]

    The Pareto frontier admits a closed-form solution under the LQR formalism, providing an exact reference for policy-level evaluation

    This choice is consistent with the quadratic structure of the cost, enabling exact recovery of q and r from the estimated utility weight matrix xW . The Pareto frontier admits a closed-form solution under the LQR formalism, providing an exact reference for policy-level evaluat...

    2019

  12. [20]

    One common practical refinement to improve the overall performance is to employ an ensemble of predictors trained from randomized initializations

    can be employed with good empirical performance, though with convergence guarantees only to local optima. One common practical refinement to improve the overall performance is to employ an ensemble of predictors trained from randomized initializations. When using such an ensem...

    2000

  13. [21]

    models. Considering a train dataset of N“2000 randomly sampled trajectory pairs, each labeled once, in the GridWorld environment, we adapted the dataset to fit the representational capacity of each model, by eitherdiscardingortransformingdata. This corresponds to discarding bo...

    2000

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.