{"id":"95b2064f-c7b3-4732-9bde-f19af4f4e0f3","arxiv_id":"2506.09183","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task loss mixing rating classification, regression, and learned uncertainty weights improves reward learning from ratings over a classification-only baseline in synthetic control tasks.","lead":"This paper combines classification and regression objectives, weighted by learned uncertainties, to infer rewards from human ratings for reinforcement learning. The combined method beats a classification-only baseline on synthetic ratings in six control tasks, but no real human raters were used.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic ratings are a noiseless, hand-thresholded quantization of true returns; without human-subject validation, the claimed mimicry of human decision-making and the empirical advantage over RbRL are unsubstantiated.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the synthetic rating generation process. The central claim of the paper is that the proposed multi-task objective, with its learnable uncertainty weights, better captures human decision-making and consistently outperforms the classification-only RbRL baseline. For this claim to hold, the ratings used in evaluation must be a faithful proxy for human ratings. The paper instead generates ratings by manually defining reward boundaries, which amounts to a noiseless, deterministic quantization of the true environment return. This removes precisely the human uncertainty and subjectivity that the method is designed to model, so the experiments cannot validate the motivating claim. The load-bearing nature of this concern is clear: if the synthetic ratings are not human-like, then even a perfect experimental result says nothing about the method's value for RLHF. A human-subject study would settle whether the concern lands. The reader's conditional verdict is appropriate because the defect is fixable: add human-rating experiments, tone down the PPO overclaim, clarify the ambiguous notation in Eq. (3), and provide rating-generation details. Our assessment therefore does not move the verdict; it reinforces the conditional accept with the same core caveat. A secondary reproducibility issue, the malformed summation in Eq. (3), should also be corrected, but it is not the primary threat to the central claim.","tokens_in":7930,"tokens_out":8791,"duration_ms":93186,"concrete_test":"Conduct a human-rating study on at least one environment (e.g., Cartpole-balance) using the same rollout segments and rating scales (n=3 and n=5) as in Section 5. Collect ratings from multiple human raters, then train the proposed multi-task method and the classification-only RbRL baseline on these human ratings under the protocol described in Section 5. Compare average return over 10 independent runs. If the multi-task loss does not outperform the classification-only loss on human ratings, the central claim about mimicking human decision-making is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 states that 'The rating data are generated synthetically by manually defining the reward boundaries for each environment.' This yields ratings that are a deterministic, noiseless quantization of the true normalized return using hand-chosen thresholds. The paper's motivating claim is to 'capture the inherent uncertainty in human decision-making' (abstract) and to model humans as integrating classification and regression strategies. Real human ratings are noisy, subjective, and not a deterministic function of the true return; the synthetic generator injects none of that uncertainty. Consequently, Figures 2 and 3 only demonstrate that the proposed loss helps recover a reward from a known quantized oracle signal, not that it mimics human rating behavior or that the learned uncertainty weights reflect human inconsistency. The manual boundaries also create a potential shortcut: if the same boundaries used to generate labels are used as the classifier's R̄i in Eq. (2), the classification subtask is artificially easy. Additionally, Eq. (3) is not well-defined as written: 'Ri(σ) = log(1 + α Σ_{t=0}^k yi)' sums over t a quantity yi that does not depend on t, making the regression target ambiguous and its scale mismatched with the normalized classifier output. The results section further overclaims PPO performance, first stating 'exceeds PPO performance given a suitable number of rating classes' and then 'surpasses PPO across all environments.' The load-bearing empirical claim therefore rests on an unvalidated proxy for human ratings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task reward learning framework for reinforcement learning from human ratings. The method trains a reward predictor by jointly optimizing a classification loss (predicting discrete rating classes) and a regression loss (predicting a smoothed continuous reward target derived from ratings), with learnable uncertainty-based weights that balance the two objectives. The authors evaluate the method on six DeepMind Control environments with synthetic ratings, comparing against standard PPO and rating-based RL (RbRL), and report that the proposed method consistently outperforms RbRL and can sometimes surpass PPO. An ablation study compares the full method against fixed-equal-weight, classification-only, and regression-only variants.","tokens_in":8118,"tokens_out":5178,"duration_ms":54451,"significance":"If the central empirical claim held, the paper would provide a useful incremental extension of rating-based reinforcement learning: adding an uncertainty-weighted regression term to a rating classifier is a natural idea, and the adaptive weighting mechanism is well motivated by the multi-task learning literature. The paper includes ablations across several environments, multiple rating-class counts, and 10 independent runs, which are strengths. However, the empirical validation rests entirely on synthetic ratings generated as a deterministic, noiseless quantization of true returns using manually chosen boundaries. This does not test the paper's stated goal of capturing the inherent uncertainty in human decision-making, and it leaves open the possibility that the reported gains are artifacts of the synthetic protocol. The contribution is therefore promising but not yet convincingly established.","major_comments":[{"comment":"The rating data are generated synthetically by 'manually defining the reward boundaries for each environment,' which yields a deterministic, noiseless quantization of the true normalized return with no human noise or inconsistency. The abstract and introduction claim the method 'captures the inherent uncertainty in human decision-making,' but the experiments never expose the reward model to such uncertainty. Consequently, Figures 2 and 3 demonstrate only that the multi-task loss can recover a reward from a known quantized oracle signal, not that it mimics human rating behavior. Please add experiments with real human raters or, at minimum, synthetic ratings with calibrated noise and a sensitivity analysis over the manual boundary choices, and temper the claims about mimicking human decision-making.","section":"Section 5"},{"comment":"Eq. (3) defines Ri(σ) = log(1 + α Σ_{t=0}^k yi), but yi is not indexed by t, so the summation is ambiguous: if the rating is assigned to the whole segment, the sum should not appear; if ratings are assigned per timestep, the notation should be y_t or similar. Moreover, the target Ri(σ) is not normalized to [0,1], while the predicted return R̂(σ) is normalized, so the MSE in Eq. (4) compares quantities on different scales. Please define the target clearly and specify the normalization used before computing Lreg.","section":"Section 4.1, Eq. (3)"},{"comment":"The results section contains a self-contradictory pair of sentences: it first says the method 'exceeds PPO performance given a suitable number of rating classes' and then claims that 'even in reward-free environments, our method surpasses PPO across all environments by adapting to different numbers of rating classes.' These statements are mutually inconsistent about whether PPO is surpassed in all environments or only for some rating-class counts. Please correct the text and state precisely what Figures 2 and 3 show, including which configurations fail to surpass PPO.","section":"Section 5"},{"comment":"The synthetic ratings are generated using manually defined reward boundaries, while Eq. (2) defines the classifier's rating boundaries R̄i as derived from the human label distribution. It is unclear whether the same manually defined boundaries used for generating labels are also used as R̄i when training the classifier. If they are, the classification subtask is given the ground-truth thresholds and may be artificially easy, which could inflate the reported advantage over RbRL. Please clarify how R̄i is set in the experiments and, if the boundaries are shared, analyze sensitivity to the specific boundary choices.","section":"Section 5, Eq. (2)"},{"comment":"The experimental description is missing essential details needed to reproduce the results: the architecture of the reward predictor, the number of reward-model updates per policy step, how and when the 2,000 synthetic ratings are collected relative to policy training, and the exact procedure for normalizing predicted returns. Please provide a complete specification of the training pipeline, including all reward-model hyperparameters, so that the empirical claims can be independently checked.","section":"Section 5 and Appendix A"}],"minor_comments":[{"comment":"The sentence 'Where yi ∈ n denotes the human-provided rating class' should be written as 'yi ∈ {0, 1, ..., n−1}' or similar; the notation 'n' alone is not a set.","section":"Section 4.1"},{"comment":"The phrase 'A similar ideal applies' should read 'A similar idea applies.'","section":"Section 4, after Eq. (5)"},{"comment":"The heading 'Acknowledgment' is misspelled; it should be 'Acknowledgments' or 'Acknowledgement' depending on the journal style.","section":"Acknowledgment"},{"comment":"The appendix provides only PPO hyperparameters; please also state the reward-model architecture, optimizer, learning-rate schedule, and number of training iterations, as these are needed for reproducibility.","section":"Appendix A"},{"comment":"Several equations and inline math have spacing artifacts (e.g., 'M DP\\R', 'λ cls', 'logλ cls'); please format them consistently.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is accepted to an ICML workshop and appears to be a short contribution. For a journal venue, the central claim about mimicking human decision-making requires validation with real human ratings or a much more careful synthetic protocol that injects realistic noise. The self-contradictory PPO claims and the ambiguity in Eq. (3) are fixable but must be addressed before the paper can be considered further. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you. This is a workshop paper that takes White et al.'s rating-based RL classifier and adds a regression head on a log-scaled version of the rating, with Kendall et al.'s uncertainty weights to combine the two losses. That specific combination is genuinely new relative to the citations, and the idea is plausible: human rating behavior has both categorical and continuous aspects, and the multi-task loss is a sensible way to let the data decide which matters more. The ablation study is solid for a six-page paper, and the experimental sweep across five rating-class counts is more than many workshop papers bother with.\n\nThe main problem is the evaluation. Every 'human rating' in the paper is synthetic: boundaries are manually chosen per environment, and ratings are generated by thresholding the true normalized return. So the data are a noiseless quantization of exactly the signal the reward model is supposed to recover. That does not validate the paper's claimed 'capture the inherent uncertainty in human decision-making.' It shows the method can invert a known quantized oracle. There is no human-subject evidence, and the manual boundary choice could also make the classification subtask artificially easy if the same boundaries are used in Eq. (2) as the label-generating thresholds. The paper doesn't say.\n\nThere are also two smaller issues. Eq. (3) as printed is not well-defined: the sum over t contains a quantity yi that has no t dependence, and the scale of the regression target is not matched to the normalized classifier output. And the results section contains a direct contradiction, saying the method 'exceeds PPO performance given a suitable number of rating classes' and then, one sentence later, 'surpasses PPO across all environments'; the abstract only claims 'in some cases.' RbRL2.0, a close baseline from the same group, is cited but never benchmarked. No code is released.\n\nNone of this kills the core idea. The method is a straightforward extension, the experiments probably support the modest claim against RbRL, and the fixes are concrete: correct Eq. (3), benchmark RbRL2.0, add at least a small human-rating study or clearly frame the results as synthetic-only, and reconcile the PPO sentence. I'd send it to peer review for a workshop, but with a request for major revision. If you work on rating-based RLHF, it's worth a skim; I wouldn't build on the synthetic results.","headline":"Plausible extension of rating-based reward learning, but synthetic-only ratings and an overclaimed PPO comparison keep it at workshop level.","tokens_in":8752,"tokens_out":3744,"would_cite":true,"duration_ms":37088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human ratings carry both category and magnitude; reward learning should use both.","keywords":["reinforcement learning from human feedback","rating-based reinforcement learning","multi-task reward learning","reward inference","uncertainty-weighted loss","classification and regression","synthetic human ratings","DeepMind Control Suite"],"falsifier":"Collect real human ratings on the same six DeepMind Control environments under the same 2 to 6 class rubrics, fix the same hyperparameters, and compare mean returns. If the multi-task method no longer consistently beats classification-only rating-based reinforcement learning, or if PPO trained on the true reward wins across the board, the synthetic rating proxy—not the combined loss—was the source of the reported improvement.","tokens_in":7612,"feed_emoji":"⭐","tokens_out":6136,"duration_ms":57205,"temperature":0.7,"pith_summary":"Reinforcement learning from human feedback typically reduces ratings to a classification task: predict which rating bucket a behavior falls into. This paper argues that humans also judge magnitude—how far a behavior is from a quality threshold—so reward learning should be a multi-task problem. It proposes training the reward predictor with both a classification loss and a regression loss, where discrete ratings are first mapped through a log-scaled formula to continuous reward targets, and the two losses are combined with learnable uncertainty weights. Experiments on six continuous-control environments with synthetic human ratings show the combined objective consistently beats the classification-only rating baseline and, for suitable numbers of rating classes, can exceed PPO trained on the true environment reward. If this holds with real raters, it would make rating feedback a more informative and cheaper signal for aligning agents in reward-free settings.","feed_headline":"Multi-task rating loss beats rating-only RL and sometimes PPO","feed_subtitle":"Adding uncertainty-weighted regression to a rating classifier lifts returns across six control tasks and rating granularities.","key_machinery":"The load-bearing object is the multi-task loss of Eq. (5), built from a rating-induced reward assignment $R_i(\\sigma) = \\log(1 + \\alpha \\sum_t y_i)$ and the classification probability $Q_\\sigma(i)$ from the base rating method. The log-scaled assignment turns discrete rating classes into continuous, smoothly varying regression targets while keeping large ratings from producing unstable gradients, which matters because early random-policy data are dominated by low ratings. The loss combines the classifier's cross-entropy with mean-squared error against those targets and weights each term by an inverse learned variance, so the model adaptively emphasizes whichever view of the ratings—category or magnitude—is currently more certain. The $\\log \\lambda$ terms prevent the weights from collapsing to zero, which would otherwise let the model ignore both losses.","core_discovery":"The central claim is that the ordinal and continuous information in human ratings is thrown away when rewards are learned only through classification. The paper proposes a rating-induced reward assignment $R_i(\\sigma) = \\log(1 + \\alpha \\sum_{t} y_i)$ that converts discrete rating classes into smoothed continuous reward labels, and a multi-task loss $L = \\frac{1}{2\\lambda_{\\mathrm{cls}}^2} L_{\\mathrm{CE}} + \\log \\lambda_{\\mathrm{cls}} + \\frac{1}{2\\lambda_{\\mathrm{reg}}^2} L_{\\mathrm{reg}} + \\log \\lambda_{\\mathrm{reg}}$ that trains the reward predictor to both classify the rating class and regress the continuous value, with $\\lambda_{\\mathrm{cls}}$ and $\\lambda_{\\mathrm{reg}}$ learned from data. The learnable weights let the model lean on classification when that signal is confident and on regression when the continuous value is more informative. Across six DeepMind Control environments and 2 to 6 rating classes, this joint objective consistently outperforms the classification-only rating-based reinforcement learning baseline, and in several configurations it matches or beats PPO, which is trained on the environment's handcrafted reward.","pith_inferences":["If real human rating data confirm the synthetic results, the method implies that raters' numerical scores carry usable magnitude information beyond category labels, which would justify collecting scalar ratings instead of only pairwise preferences.","A natural test is to rerun the six environments with genuine human raters; if the margin over the classification-only baseline shrinks or disappears, the manually defined reward boundaries used to generate synthetic ratings were the driver, not the multi-task loss.","Because the uncertainty weights are trained per task, the same framework could be extended to switch emphasis within a trajectory or to weight ratings by annotator confidence, though the paper itself tests only per-environment weighting.","The paper's own impact statement notes that biased human ratings could be reinforced; a concrete follow-up would be to audit the learned reward for rating-bias amplification before deployment in sensitive settings."],"forward_implications":["Adding a regression term to the rating classifier raises mean returns relative to classification-only rating-based reinforcement learning in all six environments tested.","With a suitable number of rating classes, the reward-free method can match or exceed PPO, which receives the true environment reward, across every environment in the study.","Learnable uncertainty weights outperform fixed equal weighting and either objective alone, indicating that the balance between classification and regression should change during training.","The log-scaled mapping from ratings to rewards makes training stable enough to learn from scratch on data dominated by low-rating segments, since the agent starts from a random policy.","Across the tested 2 to 6 rating-class configurations, the improvement is consistent rather than limited to one rating granularity."],"supporting_citations":[{"why":"Establishes reward learning from human preferences with a classifier, the paradigm this work extends to multi-task rating learning.","marker":"Christiano et al., 2017"},{"why":"Provides the classification-only rating-based reinforcement learning baseline and the rating-class probability formulation that the multi-task loss builds on.","marker":"White et al., 2024"},{"why":"Supplies the homoscedastic uncertainty weighting scheme that the proposed multi-task loss adapts for classification and regression.","marker":"Kendall et al., 2018"},{"why":"Provides the PPO algorithm used as the RL backbone and as the true-reward baseline that the method sometimes surpasses.","marker":"Schulman et al., 2017"},{"why":"Supplies the six DeepMind Control environments used for all experiments and ablations.","marker":"Tassa et al., 2018a"},{"why":"Defines the Markov decision process and reinforcement learning formalism that the reward-free MDP setting is built on.","marker":"Sutton & Barto, 1998"}],"fun_headline_variants":["Joint rating classification and regression outperforms rating-only RL","Uncertainty-weighted multi-task reward learning lifts control returns","Multi-task rating loss beats rating-only baseline, sometimes PPO","Learnable weighting of classification and regression improves rating-based RL","Multi-task reward from human ratings beats single-task baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All ratings are synthetic, generated by manually defined reward boundaries per environment; the claim that the method mimics human decision-making and captures human uncertainty assumes those boundaries faithfully model how real people assign rating categories and values.","fun_headline_variants_meta":{"raw":{"variants":["Joint rating classification and regression outperforms rating-only RL","Uncertainty-weighted multi-task reward learning lifts control returns","Multi-task rating loss beats rating-only baseline, sometimes PPO","Learnable weighting of classification and regression improves rating-based RL","Multi-task reward from human ratings beats single-task baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3625,"prompt_tokens":933,"completion_tokens":2692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2611}},"tokens_in":549,"tokens_out":2692,"duration_ms":20277,"temperature":1.0,"reasoning_tokens":2611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:54:50.432792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real human ratings on the same six DeepMind Control environments under the same 2 to 6 class rubrics, fix the same hyperparameters, and compare mean returns. If the multi-task method no longer consistently beats classification-only rating-based reinforcement learning, or if PPO trained on the true reward wins across the board, the synthetic rating proxy—not the combined loss—was the source of the reported improvement.","supporting_citations":[{"cited_title":"J., Waytowich, N., and Cao, Y","cited_arxiv_id":null,"evidence_quote":"Provides the classification-only rating-based reinforcement learning baseline and the rating-class probability formulation that the multi-task loss builds on."},{"cited_title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","cited_arxiv_id":null,"evidence_quote":"Supplies the homoscedastic uncertainty weighting scheme that the proposed multi-task loss adapts for classification and regression."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Markov decision process and reinforcement learning formalism that the reward-free MDP setting is built on."}],"review_version":1}