{"id":"6dc0b05f-aa95-4611-b217-f3a07acd061b","arxiv_id":"2509.10515","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"UAPO decomposes pairwise preference loss into two anchor-based terms, enabling offline LLM alignment with unpaired feedback at competitive benchmark scores.","lead":"This paper introduces UAPO, a training method that adds a learned 'utility anchor' per prompt to preference optimization losses, allowing language models to be aligned with feedback that is not necessarily organized into winner-loser pairs. The paper claims this makes alignment training more robust and data-efficient than standard DPO-style methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8) claims an exact rewrite of the BT log-likelihood but Eq. (7) shows the UAPO objective is only a strict lower bound; the unpaired-data decomposition (Eq. 11) and the theoretical equivalence rest on this false equality.","rationale":"The most load-bearing weakness is not the anchor parameterization, but the false equality in Eq. (8). The paper's central claim—that UAPO enables unpaired preference optimization while preserving the BT-inspired theoretical justification—depends on the exact decomposition of the pairwise log-likelihood into two sigmoid terms. Eq. (7) itself proves that this decomposition is only a strict lower bound. Consequently, Eq. (11), which generalizes to multiple winners/losers by treating the two terms independently, is not a valid rewrite of the BT objective. Appendix B confirms the issue by calling Eq. (9) a lower-bound optimization. This is an internal inconsistency, not merely a departure from consensus. It does not necessarily invalidate the empirical method: optimizing a lower bound can still be a reasonable surrogate, and the experiments may stand. However, the paper's stated theoretical motivation, the claimed equivalence to BT modeling, and the derivation of the unpaired objective are all undermined. This reinforces the reader's CONDITIONAL verdict rather than changing it: the method is plausibly useful, but the key theoretical derivation must be corrected and the unpaired objective re-derived or re-justified before the claims are fully supported.","tokens_in":22203,"tokens_out":4907,"duration_ms":56966,"concrete_test":"Analytic check: with r_w−r_⊥ = 1 and r_⊥−r_l = −1, the left side of Eq. (8) is log σ(1) + log σ(−1) ≈ −1.1269, while the right side is log σ(0) ≈ −0.6931. More generally, for finite a,b, σ(a)σ(b) < σ(a+b), so Eq. (8) cannot hold. If the authors revise Eq. (8) to a ≤ and re-derive Eq. (11) without relying on the equality assumption, the paper's theoretical claims and the unpaired-data derivation must be updated; the surrogate objective can then be evaluated on its own merits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central derivation is invalid at Eq. (8). From Eq. (7), log σ(r_w−r_⊥) + log σ(r_⊥−r_l) ≤ log σ(r_w−r_l). This inequality is strict for every finite reward: σ(a)σ(b) = 1/((1+e^{-a})(1+e^{-b})) < 1/(1+e^{-a-b}) = σ(a+b). Eq. (8) nevertheless states equality, calling the two-term UAPO loss a rewrite of the BT log-likelihood. The objective in Eq. (9) is therefore not equivalent to BT preference modeling; it is a lower-bound surrogate. This matters because the unpaired-data extension in Eq. (11) is justified from the 'separate form' of Eq. (8) (Appendix A), and because §2.4 advertises consistency with BT-style optimization. The appendix itself later calls Eq. (9) a 'lower bound optimization' (Appendix B), so the main text is internally inconsistent: the claimed theoretical equivalence is contradicted within the same manuscript. The empirical results may still hold for the surrogate objective, but the theoretical grounding and the derivation of the unpaired objective are unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Preference Optimization with Utility Anchor (UAPO), an offline preference-optimization framework that replaces the Bradley-Terry pairwise log-likelihood with two sigmoid terms involving a learnable \"utility anchor\" r(x, y⊥). The anchor is intended to model annotation uncertainty and to allow training from unpaired or multiple winning/losing responses. The authors instantiate UAPO on DPO and SimPO (UAPO and SimUAPO), provide experiments on Mistral-7B, Llama-3-8B, and Gemma-2-9B across AlpacaEval 2, Arena-Hard, MT-Bench, RewardBench, and downstream tasks, and report competitive or improved performance with better robustness to hyperparameter changes and preference noise. The paper also claims a theoretical connection to pessimistic RL via an uncertainty penalty.","tokens_in":22442,"tokens_out":4393,"duration_ms":51796,"significance":"If the framework were rigorously established, it would be a practically useful contribution: offline preference optimization from unpaired data would improve data efficiency, and the anchor mechanism could provide robustness to label noise and distribution shift. The empirical evaluation is broad, including multiple base models, standard alignment benchmarks, OOD reward benchmarks, KL-divergence analysis, robustness to preference corruption, and downstream-task retention. The authors are also candid about MT-Bench instability and include ablations on the constant reward anchor. However, the central theoretical derivation contains a load-bearing error: Eq. (8) asserts an equality that Eq. (7) itself shows to be false, and the unpaired-data extension in Eq. (11) is not actually derived. The empirical results may hold for the surrogate objective, but the paper's main conceptual claims about equivalence to BT modeling and about the anchor's uncertainty semantics are not supported as written.","major_comments":[{"comment":"The claim that Eq. (8) is a rewrite of the BT log-likelihood is incorrect. Eq. (7) correctly establishes log σ(r_w − r_⊥) + log σ(r_⊥ − r_l) ≤ log σ(r_w − r_l), with strict inequality for every finite reward. Therefore the UAPO objective in Eq. (9) is not equivalent to BT preference modeling; it is a surrogate obtained by splitting the pairwise margin. This error is internal to the manuscript: Appendix B later explicitly treats Eq. (9) as a lower-bound optimization, while §2.3 and the abstract advertise consistency with BT-style optimization. Moreover, if Eq. (9) is interpreted as a loss, the inequality implies L_UAPO ≥ L_BT, so the paper's language about a \"lower bound\" is also misleading. This invalidates the theoretical grounding of the unpaired-data derivation, which relies on the \"separate form\" of Eq. (8).","section":"§2.3, Eq. (7)–(8)"},{"comment":"The extension to multiple winning and losing responses is not derived. Starting from Eq. (9), for a single pair one has L_w = −log σ(r(x,y_w) − r(x,y_⊥)). For multiple winners, the natural softmax-type generalization would place each winning response in a denominator with the anchor and all other candidates; the formula in Eq. (11) instead sums over all winning responses in the denominator of each term, which double-counts the target response and is not shown to follow from Eq. (9). The single L_l term is also asymmetric. Appendix A simply states the generalization without a proof. Since learning from unpaired data is a central advertised contribution, this step needs a correct, explicit derivation or a clearly stated modeling assumption.","section":"§2.3, Eq. (11) and Appendix A"},{"comment":"The interpretation of Figure 1(a) as showing that the utility anchor \"effectively classifies\" responses is circular. The loss in Eq. (9) is minimized by making σ(r_w − r_⊥) and σ(r_⊥ − r_l) large, i.e., by pushing the anchor between winning and losing rewards. That the anchor lies between them during training is therefore a consequence of the objective, not an empirical discovery about the anchor's learned representation. The stronger claim that the anchor captures prompt-dependent annotation uncertainty is an assumption that is not tested. Appendix C implements the anchor as a linear function of last-layer hidden states, but no ablation against other parameterizations (e.g., MLP, scalar, or non-parametric anchor) is reported, so the uncertainty-aware interpretation is under-supported.","section":"§3.3, Figure 1"},{"comment":"The theoretical analysis connecting UAPO to pessimistic RL is not rigorous. Eq. (14) writes a gradient with respect to π_θ while treating u_ϕ as both a fixed penalty and a function of π_θ; the derivation is not given. Eq. (15) is asserted as the equivalent policy likelihood, but the equivalence is not proven. More importantly, because Eq. (8) is invalid, the claimed consistency with BT-style preference optimization and the uncertainty-penalty interpretation do not follow. The authors should either provide a careful derivation of Eq. (15) or state explicitly that this is an analogy, not a theorem.","section":"§2.4, Eq. (13)–(15)"}],"minor_comments":[{"comment":"The sentence \"Since the sigmoid function is monotonically increasing\" does not by itself imply the inequality; the inequality follows from σ(a)σ(b) ≤ σ(a+b). Please rephrase to state the actual property used.","section":"§2.3, Eq. (7)"},{"comment":"The notation is inconsistent: Eq. (6) defines r(x, y_⊥) using a dummy token y_⊥, but Appendix C defines log π(x, y_⊥) as a product over prompt-token hidden states, which is not a single dummy token. Clarify what y_⊥ represents.","section":"Appendix C"},{"comment":"Figure 1 is referenced in the discussion of reward comparisons before it is introduced. Reorder or make the cross-reference explicit.","section":"§2.2"},{"comment":"Minor typos: \"demonstate\" (Section 1) and \"Goodhard's law\" should be \"Goodhart's law.\"","section":"Introduction"},{"comment":"The term \"unpaired data\" is used loosely. The reconstructed multiple datasets still contain a designated winning response and three losing responses per prompt; they are not unpaired in the sense of having only one response per prompt. Clarify the intended meaning.","section":"§3.1 and §3.2"},{"comment":"The dataset construction section says five responses are generated and one is selected as winning; the main text says four responses and three losing. The numbers should be reconciled (main text says \"four candidate responses... one winning and three losing,\" while Appendix D says five).","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The central inequality error is serious and is admitted, in effect, by Appendix B, so the authors have the raw material to fix the framing. I would be willing to reconsider after a revision that (1) replaces the equality claim with an explicit surrogate/lower-bound statement, (2) re-derives or substantially revises the unpaired-data objective, and (3) either validates or softens the uncertainty-anchor interpretation. The empirical contribution is solid enough to merit this chance rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: UAPO replaces the fixed z0 in KTO and the fixed margin in SimPO with a learned, prompt-dependent anchor. That is a real extension, and the experiments show gains on several benchmarks, especially when you have multiple winners and losers per prompt. The multi-dataset objective in Eq. (11) is new. If you work on unpaired preference optimization, this is worth reading.\n\nThe central problem is Eq. (8). The paper writes the two-sigmoid sum as an exact rewrite of the BT log-likelihood, but Eq. (7) immediately above shows it is only a lower bound. The inequality is strict for finite rewards. The appendix later calls it a lower bound optimization, so the main text contradicts itself. This matters: the unpaired objective in Eq. (11) is justified from the 'separate form' of Eq. (8), and Section 2.4 claims consistency with BT modeling. The theoretical grounding for unpaired training is unsupported as written.\n\nThe anchor itself is also a bit circular. The loss pushes the anchor between winning and losing rewards, so Figure 1 just shows the objective doing its job. The linear parameterization is not compared to alternatives, and the paper never validates that it captures annotation uncertainty rather than just being a flexible threshold. The missing KTO baseline in the unpaired experiments is a real gap, since that is the main existing method that handles unpaired data. The noise-robustness result in Table 6 is mixed on Llama-3: the LC drop is smaller with SimUAPO but the WR drop is larger, so the robustness claim is not clean. And the main tables lack error bars.\n\nWhat the paper does well: it identifies a reasonable gap in the literature, proposes a concrete fix, and tests it across three models and several benchmarks. The empirical results, especially on AlpacaEval 2 and Arena-Hard, are suggestive. The method is simple and could be a useful baseline.\n\nMy take: the method is plausible but the theory as written is wrong. This is a conditional paper, not a reject. The authors can fix Eq. (8) to state a lower-bound objective, add KTO, add error bars, and ablate the anchor parameterization. If they do that, it becomes a solid contribution. I'd send it to peer review.","headline":"A useful empirical method with a mis-stated theoretical claim; the anchor idea is worth a look, but Eq. (8) is a false equality that undermines the paper's main narrative.","tokens_in":23006,"tokens_out":6691,"would_cite":true,"duration_ms":71160,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned, prompt-dependent utility anchor lets offline preference optimization train on unpaired winning and losing responses while keeping competitive alignment performance.","keywords":["preference optimization","utility anchor","unpaired data","Bradley-Terry model","LLM alignment","uncertainty penalty","direct preference optimization","offline learning"],"falsifier":"Compute, on held-out prompts, the correlation between the learned anchor and an oracle uncertainty threshold obtained from a known latent reward with controlled label noise; a weak or negative correlation would falsify the claim that the anchor estimates annotation uncertainty. A second check would replace the linear anchor with a per-prompt learned scalar and observe whether the unpaired-training benefit disappears.","tokens_in":1621,"feed_emoji":"🎯","tokens_out":2183,"duration_ms":96441,"temperature":0.7,"pith_summary":"This paper argues that the pairwise assumption in offline preference optimization, encoded by the Bradley-Terry model, is not necessary. It proposes UAPO, which replaces a single winning-versus-losing comparison with two comparisons: winning response versus a learned utility anchor, and anchor versus losing response. Because the anchor is prompt-dependent, the objective can be evaluated on unpaired preferred and dispreferred responses, so alignment training no longer requires matched pairs. The paper shows that UAPO wraps existing methods such as DPO and SimPO, improves stability under noisy preference labels and across hyperparameter settings, and preserves downstream knowledge. If correct, this would make preference data collection cheaper and alignment training more flexible.","feed_headline":"Utility anchor lets LLM alignment train without paired responses","feed_subtitle":"Splitting each preference into two comparisons around a learned threshold lets unpaired feedback drive alignment.","key_machinery":"The utility anchor r(x, y_anchor) is the load-bearing object: a learnable prompt-dependent reward threshold, implemented as a linear map over the prompt's last-layer hidden states, plus a constant offset. It splits the pairwise preference signal into two sigmoid comparisons, winning-above-anchor and anchor-above-losing, which is what makes unpaired training possible. The two-term loss in Eq. (9) is the mechanism that carries the argument, and the multi-response generalization in Eq. (11) is what lets the method exploit imbalance between preferred and dispreferred responses. The pessimistic-RL interpretation of the anchor as an uncertainty penalty connects the construction to overoptimization","core_discovery":"UAPO's central move is to replace the Bradley-Terry log-likelihood log sigma(r(x,y_w) - r(x,y_l)) with log sigma(r(x,y_w) - r(x,y_anchor)) + log sigma(r(x,y_anchor) - r(x,y_l)), where the anchor is a dummy token whose reward is the utility anchor. The anchor is a learnable, prompt-dependent reference point, implemented as a linear function of the prompt's last-layer hidden states, that is trained to sit between the winning reward and the losing reward. Because the two sigmoid terms separate wins from losses, the loss no longer needs the two responses to come from the same comparison; the paper derives a multi-response form in Eq. (11) that pools any number of preferred and dispreferred respo","pith_inferences":["Beyond the paper, the linear anchor is the least-tested piece: if annotation uncertainty is highly nonlinear in the prompt, a linear map from last-layer hidden states may cap the method's accuracy. A direct ablation against a small MLP anchor or a per-prompt learned threshold would settle this.","The anchor's role as an uncertainty penalty suggests an active-learning extension the paper does not explore: prompts whose anchor is close to the winning or losing reward are exactly the ambiguous cases worth re-labeling or sending to a stronger judge.","If the unpaired-data claim holds, preference data from naturally occurring signals, such as thumbs up/down, user edits, or model self-corrections, could be used directly without constructing synthetic losing responses, substantially broadening the data sources for alignment."],"forward_implications":["Offline preference optimization can be run on datasets with unpaired winning and losing responses, so data pipelines no longer need to force every prompt into exactly one win/lose pair.","Existing preference objectives can be converted by inserting an anchor; the paper demonstrates this for DPO, SimPO, and R-DPO, with the anchored variants matching or exceeding their base methods on instruction-following benchmarks.","The training signal becomes more stable: with 40% of win/lose labels randomly swapped, UAPO's performance drop on AlpacaEval 2 is smaller than SimPO's, and the method tolerates a wider range of the constant reward margin.","KL divergence to the reference policy stays lower and smoother during training, which the paper links to retaining general knowledge and mathematical reasoning on downstream tasks.","Because Eq. (11) pools multiple preferred and dispreferred responses, the method can use the abundant losing responses instead of discarding them, improving data utilization."],"supporting_citations":[{"why":"Supplies the Bradley-Terry pairwise model that UAPO replaces and whose assumptions the paper challenges.","marker":"(Bradley and Terry, 1952)"},{"why":"Provides the DPO reward reparameterization that UAPO uses as one instantiation of its framework.","marker":"(Rafailov et al., 2023)"},{"why":"Supplies the reference-free SimPO reward and the baseline that SimUAPO extends and compares against.","marker":"(Meng et al., 2024)"},{"why":"Provides KTO, the main prior unpaired-data preference method that UAPO contrasts with its learned anchor.","marker":"(Ethayarajh et al., 2024)"},{"why":"Provides the pessimistic-RL uncertainty-penalty view used to interpret the anchor as a lower confidence bound.","marker":"(Jin et al., 2021)"},{"why":"Documents reward overoptimization and distribution shift, the problem the anchor is designed to mitigate.","marker":"(Gao et al., 2023)"},{"why":"Supplies the cognitive evidence of risk-averse and reference-dependent human decisions that motivates the utility anchor.","marker":"(Tversky and Kahneman, 1992)"}],"fun_headline_variants":["Unpaired LLM alignment via utility anchor","Anchor splits preference, no paired data needed","UAPO: train LLM alignment without response pairs","Utility anchor unlocks unpaired preference learning","LLM alignment improved with anchor, not pairs"],"cache_read_input_tokens":24704,"weakest_assumption_plain":"The load-bearing premise is that a prompt-dependent utility anchor learned by a linear function reliably estimates the uncertainty threshold between preferred and dispreferred responses; if it does not, the two sigmoid comparisons misclassify responses and the unpaired-training claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Unpaired LLM alignment via utility anchor","Anchor splits preference, no paired data needed","UAPO: train LLM alignment without response pairs","Utility anchor unlocks unpaired preference learning","LLM alignment improved with anchor, not pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1011,"prompt_tokens":709,"completion_tokens":302,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":453,"tokens_out":302,"duration_ms":3489,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:06:04.404122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on held-out prompts, the correlation between the learned anchor and an oracle uncertainty threshold obtained from a known latent reward with controlled label noise; a weak or negative correlation would falsify the claim that the anchor estimates annotation uncertainty. A second check would replace the linear anchor with a per-prompt learned scalar and observe whether the unpaired-training benefit disappears.","supporting_citations":[],"review_version":1}