{"id":"2b30cca6-cb31-4e0d-93a3-e353b15e9125","arxiv_id":"2506.12529","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SARA extracts rewards from cosine similarity to a contrastively learned latent of preferred trajectories and outperforms or matches baselines under label noise in continuous control benchmarks.","lead":"SARA is a new method for preference-based reinforcement learning that calculates rewards by measuring how similar a behavior is to a learned summary of preferred examples. It reports more stable and competitive performance than existing methods when labels contain mistakes, and it works across offline, online, and cross-task settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's Wilcoxon p<0.01 significance claim is absent from the body and, under a six-task pairing, impossible; the 31% gain is an unweighted average over 18 baseline entries, so the central statistical claim is unsupported as stated.","rationale":"The reader's stated weakest assumption is that a single preferred-set latent z*_p sufficiently summarizes the preference criterion. That is a real methodological risk, and I agree it deserves testing. However, the most load-bearing concern about the paper's central claim is more immediate: the abstract promises a Wilcoxon p<0.01 significance result and a 31% improvement, but the body reports no significance test and no aggregation definition for the 31% figure. The table is consistent with 31% only under a generous unweighted average over all 18 baseline entries, and the natural six-task paired Wilcoxon test cannot reach p<0.01 at all. This is missing support for the exact headline claim, not a disagreement with the method. I credit the released code, the careful IQL pipeline matching, and the clear evaluation protocol as genuine supporting evidence; my concern is about the statistical reporting rather than the algorithm's internal consistency. The reader already marked the paper CONDITIONAL partly because of the missing Wilcoxon test, so my concern reinforces that verdict instead of moving it.","tokens_in":33060,"tokens_out":8583,"duration_ms":100551,"concrete_test":"Recompute the Wilcoxon signed-rank test from the per-seed evaluation returns for the 20%-noise condition, comparing SARA against each baseline, and report n, test direction, and exact p-values; also state the exact aggregation formula used for the 31% gain. If no comparison reaches p<0.01, or if the gain is about 15% against PT/PT+ADT, revise the abstract and Section 4 to report the actual significance level and per-task mean gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in the abstract—'statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01)'—is asserted but never reported in Section 4 or the appendices. The only summary statistic in the body, the 31% mean relative gain (Section 4, 'Offline experiments summary'), also lacks a stated aggregation formula. Reproducing it from Table 1 requires averaging the per-baseline relative gains over all 18 baseline entries, which gives 30.9%; the per-task mean relative gain is about 22.8%, and against the two reward-model baselines (PT and PT+ADT) it is only about 15.1%. Much of the 31% is driven by DPPO's collapse on hopper-expert, where SARA shows a 194% relative gain. More importantly, with only six benchmark tasks, a paired Wilcoxon signed-rank test cannot reach p<0.01: for n=6 the minimum two-sided exact p-value is 0.03125. The claimed significance must therefore rely on some other, unreported pooling, per-seed design, or multiple-comparison procedure. No such test, effect size, or confidence interval is provided. The core robustness claim is therefore not statistically verified as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Similarity as Reward Alignment (SARA), a contrastive framework that learns a latent representation of preferred trajectories and assigns per-timestep rewards as the cosine similarity between the encoded trajectory and a single preferred-set latent z*_p. The method is evaluated on D4RL offline control tasks (Mujoco locomotion, Franka Kitchen, and Adroit pen), with IQL as the downstream RL algorithm, against PT, PT+ADT, and DPPO baselines across several preference-set variants: human vs. script labels, inclusion vs. exclusion of neutral preferences, and a 20% label-error condition. The paper also presents three additional applications: low-quality trajectory filtering for imitation learning, cross-task preference transfer from hopper to walker2d, and online reward shaping on a Deepmind Control Suite Reacher task. The central empirical claim is that with 20% label error SARA achieves a 31% mean relative gain in policy evaluation rewards over baselines, with a statistically significant improvement asserted in the abstract (Wilcoxon signed-rank, p < 0.01).","tokens_in":33232,"tokens_out":9186,"duration_ms":109645,"significance":"If the empirical claims are fully supported, SARA would be a simple and versatile preference-based RL method with a useful property: it produces an explicit reward signal without Bradley-Terry modeling, and it can be plugged into off-the-shelf offline and online RL algorithms. The paper has notable strengths: the code is publicly released; the offline evaluation uses 8 seeds with a carefully documented evaluation protocol (Appendix E.2); several dataset variants and labeler types are considered; and the additional applications, especially the morphologically different transfer experiment, are original. I also do not see a circularity defect in the reward construction: z*_p is learned from preference data and then evaluated on downstream policy returns, which is standard generalization, and alpha is not forced to a value that trivially produces the reported results. The main weakness is that the headline statistical claims are not backed by any reported test, the 31% aggregate is sensitive to the exact aggregation formula, and a promised correlation-to-environment-reward analysis is absent from the body.","major_comments":[{"comment":"The abstract's claim of 'statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01)' is nowhere reported in Section 4 or in the appendices. With the six primary tasks in Tables 1-4, the minimum achievable two-sided exact p-value for a paired Wilcoxon signed-rank test is 0.03125, so p < 0.01 cannot be obtained from that pairing. If the test instead pools the eight tasks including Adroit, or uses per-seed measurements, the paper must state the pairing, the sample size, the test statistic, and any multiple-comparison correction. As written, the headline significance claim is unsupported.","section":"Abstract; Section 4 (Offline experiments summary)"},{"comment":"The '31% mean relative gain in policy evaluation rewards over baselines' is not reproducible as stated because no aggregation formula is given. Reconstructing from Table 1 as an unweighted average of relative gains over the 18 baseline entries gives approximately 30.9%, but this value is strongly influenced by the 194% relative gain on hopper-expert, where DPPO collapses to 28.92. Individual baseline relative gains range from about -1% to 194%, and against PT and PT+ADT alone the mean is about 15%. The paper should specify the aggregation formula, report per-task and per-baseline gains, and provide a robust summary such as the median gain or a confidence interval before claiming a 31% advantage.","section":"Section 4, 'Offline experiments summary' and Table 1"},{"comment":"The abstract promises that 'We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria' and that 'SARA computed rewards display higher correlation across noise rates compared to baselines.' No such correlation analysis appears in Section 4 or in the appendices; Figure 3 is a t-SNE visualization, not a correlation measurement. This result must either be added with a concrete metric reported across noise rates, or the claim should be removed from the abstract.","section":"Abstract; Section 4"},{"comment":"The robustness claim that 'SARA is quite consistent across dataset variants as well as performant' is supported only by a qualitative visual comparison in Figure 2. The paper does not quantify the stability of rewards or policy returns across dataset variants, nor does it test whether the differences in fluctuation between SARA and the baselines are statistically meaningful. Given that robustness to dataset variations is one of the paper's primary claims, please add a quantitative stability measure (e.g., variance or interquartile range across variant means, or a paired test across the variants) rather than relying on the vertical range in the figure.","section":"Section 4, 'Robustness to dataset variants' and Figure 2"}],"minor_comments":[{"comment":"The abstract says 'on preference data with varying realistic noise rates,' but the offline experiments include only the original data and a single 20% label-flip condition. If additional noise rates were tested, they should be reported or cited; otherwise the wording should be adjusted.","section":"Abstract"},{"comment":"The online reward-shaping experiment is presented through a qualitative Figure 5 only. Please report numerical policy returns or success metrics for the shaped versus unshaped policy, along with the number of seeds, so that the reader can assess the strength of the effect.","section":"Section 5, 'Online RL with reward shaping'"},{"comment":"The paper describes OPPO, CPL, and IPL in the related work but does not compare against them empirically. Since the abstract claims strong performance over baselines, a sentence explaining why these methods are not included as baselines would help the reader calibrate the comparison.","section":"Related work and Section 4"},{"comment":"The logistic form in Eq. (2) is described as motivation, but it is used to justify setting the reward to cos(z_t, z*_p) with alpha = 0. Please state explicitly that this is a heuristic reward and that no formal guarantee is claimed for the ordering induced by the cosine similarity, as the limitations paragraph already acknowledges.","section":"Appendix A, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a solid empirical core and a clearly described method, but the abstract overstates the statistical support. The missing Wilcoxon test, the undefined 31% aggregation, and the absent correlation analysis are all fixable with the existing data and code, so I recommend major revision rather than rejection. I would also ask the authors to be careful, in revision, not to let the single DPPO collapse on hopper-expert carry the aggregate improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: SARA is a sound, simple idea—learn a set-level latent for preferred trajectories and use cosine similarity as reward—but the abstract overclaims what the paper demonstrates. The Wilcoxon p<0.01 claim never appears in the body, and with six tasks it cannot be true for a paired test as stated.\n\nWhat's genuinely new: I don't know another PbRL paper that uses a set-contrastive transformer to get one preferred-set latent and then derives a dense reward from cosine similarity. That's a real algorithmic configuration, distinct from PT, DPPO, and OPPO. It also side-steps Bradley-Terry assumptions, which is defensible given the label-noise literature. The offline setup is careful: 8 seeds, same seeding for encoder and policy, evaluation protocol described in detail (running mean over 80 episodes, total variance decomposition), and code is released. The PT+ADT baseline is a fair addition. The t-SNE filtering analysis and the online shaping toy are genuinely illustrative, though qualitative.\n\nSoft spots, in order of size. First, the headline statistical claim is absent. I read all of Section 4 and the appendices: no Wilcoxon test reported, no effect size, no confidence interval. The abstract's 'p<0.01' is unverifiable. Second, the 31% mean relative gain looks like an average over 18 baseline entries. Recomputing from Table 1, the per-task average is about 23%, and against PT and PT+ADT alone only about 15%. Much of the gain comes from DPPO collapsing on hopper-expert, which is a real result but shouldn't be dressed up as a blanket robustness win. Third, the abstract promises a 'correlation to environment rewards' analysis; it never appears in the text. Either put it in or drop the claim. Fourth, the Adroit tables are only in the appendix after the text says results are 'high variance and quite similar across models'—a mild selection concern, but the appendix tables confirm the point, so I don't weigh it heavily.\n\nThe theory appendix is explicitly motivational, not a proof, and the authors say so in the limitations section. That's honest.\n\nWho is this for: anyone working on preference-based RL or reward learning from noisy human labels. The central idea is worth following up. It deserves a serious referee, but the referee should push the authors to actually run and report the significance test, define the aggregation, and make good on the correlation claim.","headline":"SARA's core idea is solid and the offline setup is careful, but the abstract claims statistics the paper never reports, so the headline robustness result is not yet backed.","tokens_in":33824,"tokens_out":2325,"would_cite":false,"duration_ms":28524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SARA claims that a single latent vector for the preferred set, used as a cosine-similarity anchor, is enough to compute RL rewards that stay reliable under high label noise.","keywords":["preference-based reinforcement learning","contrastive learning","reward inference","cosine similarity reward","label noise robustness","offline reinforcement learning","reward shaping","preference transfer"],"falsifier":"Train SARA on a synthetic preference dataset whose preferred set has two equally large, widely separated modes, such as clockwise and counterclockwise trajectories to the same target, then measure whether the cosine reward to a single $z^*_p$ still ranks both modes above non-preferred trajectories; if the average latent falls between the modes and both modes receive low similarity, the central claim fails. A quantitative version compares the correlation between SARA reward and true environment reward against baselines on this bimodal set, and checks whether the correlation drops below the single-mode baseline.","tokens_in":32766,"feed_emoji":"🤖","tokens_out":5758,"duration_ms":64498,"temperature":0.7,"pith_summary":"The paper tries to establish that a preference signal for reinforcement learning can be carried by a single learned vector: the latent representation of the whole preferred set. Instead of fitting a Bradley-Terry model to pairwise comparisons, SARA trains a contrastive transformer encoder to pull preferred trajectories together and push non-preferred ones apart, then defines the reward at every timestep as the cosine similarity between the encoded trajectory and that preferred-set latent. On continuous-control offline RL benchmarks with 20% of preference labels randomly flipped, the paper reports a 31% average improvement in policy evaluation rewards over three baselines, with lower variance across dataset variants. It also argues the same reward can filter low-quality trajectories, transfer preferences between agents with different morphologies, and shape rewards in online RL.","feed_headline":"One latent drives RL rewards that beat noisy-preference baselines","feed_subtitle":"Cosine similarity to a learned preferred-set latent outperforms Bradley-Terry baselines when 20% of labels are wrong.","key_machinery":"The load-bearing object is the preferred-set latent $z^*_p$. A first transformer encodes each trajectory with positional encodings and average-pools over timesteps; the resulting per-trajectory encodings are randomly partitioned into $k$ subsets whose composition is reshuffled each epoch, and a second transformer encodes each subset into one latent. Training uses the SimCLR contrastive loss, which attracts latents of the same category and repels the two categories. At inference, the full preferred set is encoded in one forward pass to give $z^*_p$, and every timestep reward is the cosine similarity $\\cos(z_t, z^*_p)$; the paper sets the optional negative-similarity term to zero in all experiments. This machinery does the work of absorbing label noise: because subset composition is shuffled, a mislabeled trajectory cannot dominate the contrastive objective, and because only set membership matters, pairwise ranking assumptions are dropped.","core_discovery":"On its own terms, SARA claims that a reward function adequate for RL can be read off directly from representation geometry: $r_t = \\cos(z_t, z^*_p)$, where $z^*_p$ is the encoder's latent for the full set of preferred trajectories and $z_t$ is the latent of the trajectory up to time $t$. The encoder is trained with the SimCLR contrastive loss on randomly composed subsets of preferred and non-preferred trajectories, so the latent captures patterns that distinguish the categories rather than any particular pairwise ranking. The paper's central evidence is that with 20% label error, SARA rewards drive IQL policies that beat Preference Transformer, PT with adaptive denoising, and DPPO on five of six offline tasks, and that its reward correlates more strongly with the environment reward across noise rates. The same frozen latent supports trajectory filtering, preference transfer from a one-legged to a two-legged agent, and online reward shaping in a Reacher task.","pith_inferences":["If the single-latent summary holds up, preference-based RL shifts from ranking pairs to set-level prototype learning, which is a natural fit for LLM alignment data where many responses are preferred to one reference and labels are noisy.","A testable consequence the paper does not run: as the share of flipped labels in the preferred set approaches 50%, the contrastive objective has nothing consistent to align, so the reward should collapse; locating that breakpoint would calibrate the claimed robustness.","The cross-task transfer experiment suggests a more general recipe, namely mapping a simple agent's state-action dimensions into a larger agent by joint symmetry; a stronger test would use a learned correspondence instead of a crude doubling.","Because $r_t = \\cos(z_t, z^*_p)$ is a dense, bounded reward, SARA doubles as a shaped exploration bonus; combining it with curiosity or entropy bonuses could be explored in tasks where preferences are sparse."],"forward_implications":["With SARA's reward, any off-the-shelf offline RL algorithm can be trained from preferences, since a scalar reward is produced for every transition rather than only a policy objective.","At 20% label error, SARA's policy evaluation rewards average 31% higher than PT, PT+ADT, and DPPO, and its performance varies less across preference dataset variants (with or without neutral labels, human vs scripted, noisy vs clean).","The learned preferred latent is reusable: it ranks cold trajectories for filtering low-quality demonstrations and transfers hopper preferences to reward estimation for walker2d.","In online RL, adding the cosine-similarity reward to a known task reward changes the learned Reacher policy from the shortest clockwise path to the desired counterclockwise path even though the preference set only covered an easier target.","SARA accepts feedback that is not pairwise: a set of preferred trajectories alone is enough, so neutral or equal-preference labels do not need special handling beyond placing trajectories in both sets."],"supporting_citations":[{"why":"Provides the human-labeled preference datasets for the locomotion and Adroit tasks and the Preference Transformer baseline that SARA must beat.","marker":"[6]"},{"why":"Supplies the DPPO baseline and the Franka Kitchen preference datasets used in the offline experiments.","marker":"[19]"},{"why":"The SimCLR contrastive loss is the training objective that aligns preferred latents and separates non-preferred latents.","marker":"[45]"},{"why":"The transformer architecture with positional encoding is the backbone of SARA's two encoders.","marker":"[44]"},{"why":"Implicit Q-Learning (IQL) is the offline RL algorithm that consumes SARA-computed rewards in the main experiments.","marker":"[36]"},{"why":"Provides the theoretical grounding that Bradley-Terry formulations are not necessary and that per-trajectory preference prediction suffices, which motivates the cosine-reward form.","marker":"[23]"},{"why":"Supplies the evaluation protocol of averaging the last eight evaluation runs, which SARA uses to report policy rewards.","marker":"[17]"},{"why":"Motivates the robustness problem and the adaptive denoising baseline applied to the Preference Transformer.","marker":"[30]"}],"fun_headline_variants":["Latent similarity rewards beat noisy-preference baselines in RL","Cosine to learned latent: robust rewards from noisy preference labels","SARA: contrastive latent alignment for robust preference-based RL","Reward from latent similarity survives 20% label error","Simple contrastive reward robust to noisy human preferences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reward signal rests on the assumption that one vector produced by encoding the entire preferred set is an adequate summary of the preference criterion, so that cosine similarity to that vector correctly ranks any new trajectory; if the preferred set is multimodal or dominated by mislabeled trajectories, that single anchor can be biased.","fun_headline_variants_meta":{"raw":{"variants":["Latent similarity rewards beat noisy-preference baselines in RL","Cosine to learned latent: robust rewards from noisy preference labels","SARA: contrastive latent alignment for robust preference-based RL","Reward from latent similarity survives 20% label error","Simple contrastive reward robust to noisy human preferences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1272,"prompt_tokens":910,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":526,"tokens_out":362,"duration_ms":4985,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:46:36.306090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SARA on a synthetic preference dataset whose preferred set has two equally large, widely separated modes, such as clockwise and counterclockwise trajectories to the same target, then measure whether the cosine reward to a single $z^*_p$ still ranks both modes above non-preferred trajectories; if the average latent falls between the modes and both modes receive low similarity, the central claim fails. A quantitative version compares the correlation between SARA reward and true environment reward against baselines on this bimodal set, and checks whether the correlation drops below the single-mode baseline.","supporting_citations":[{"cited_title":"Direct Preference-based Policy Optimization without Reward Modeling","cited_arxiv_id":null,"evidence_quote":"Supplies the DPPO baseline and the Franka Kitchen preference datasets used in the offline experiments."},{"cited_title":"A Simple Framework for Contrastive Learning of Visual Representations","cited_arxiv_id":null,"evidence_quote":"The SimCLR contrastive loss is the training objective that aligns preferred latents and separates non-preferred latents."},{"cited_title":"Attention is All you Need","cited_arxiv_id":null,"evidence_quote":"The transformer architecture with positional encoding is the backbone of SARA's two encoders."},{"cited_title":"OFFLINE REINFORCEMENT LEARNING WITH IMPLICIT Q-LEARNING","cited_arxiv_id":null,"evidence_quote":"Implicit Q-Learning (IQL) is the offline RL algorithm that consumes SARA-computed rewards in the main experiments."},{"cited_title":"Rethinking reward modeling in preference-based large language model alignment","cited_arxiv_id":null,"evidence_quote":"Provides the theoretical grounding that Bradley-Terry formulations are not necessary and that per-trajectory preference prediction suffices, which motivates the cosine-reward form."},{"cited_title":"Bradley Knox, and Dorsa Sadigh","cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation protocol of averaging the last eight evaluation runs, which SARA uses to report policy rewards."},{"cited_title":"RIME: robust preference-based reinforcement learning with noisy preferences","cited_arxiv_id":null,"evidence_quote":"Motivates the robustness problem and the adaptive denoising baseline applied to the Preference Transformer."}],"review_version":1}