{"id":"6a11919d-4648-40c9-ba66-5fb7fc7c6ee9","arxiv_id":"2501.07755","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An empirical study proposes hyperparameter settings that improve rating-based RL on three control tasks, but the gains are measured on the same runs used to select the settings.","lead":"This paper tests many settings for rating-based reinforcement learning, a method that learns rewards from human ratings instead of hand-designed reward functions. It suggests a configuration that improves benchmark performance, but the configuration is chosen on the same tests used to judge it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central improvement claim relies on hyperparameters selected from the same environments used for evaluation; without a held-out comparison, the reported gains may be selection artifacts.","rationale":"The reader's overall REJECT verdict is appropriate, but the single most load-bearing flaw is circular evaluation: hyperparameters are chosen and evaluated on the same environments. The reader's designated weakest assumption is the synthetic-rating proxy, which is a real external-validity concern; the reader also mentions circular evaluation in the rationale, but not as the primary weakest point. I locate the decisive problem one step earlier: even within the synthetic-rater setup, the recommended settings are not validated on any independent data, so the reported improvements cannot be distinguished from selection over noisy runs. The paper does include some multi-run tables and reports standard deviations, which is creditworthy, and it explicitly identifies itself as work in progress. Those factors temper the severity but do not remove the need for a held-out evaluation. The concrete test above would settle whether the recommended configuration is genuinely better or merely the best among the tested configurations on the test set. I keep the reader's verdict unchanged because the central claim, as stated, remains unsupported without such validation.","tokens_in":9467,"tokens_out":5005,"duration_ms":53361,"concrete_test":"Pre-register the optimized hyperparameters using only Walker and Quadruped for selection, then evaluate the fixed configuration on Cheetah as a completely untouched hold-out environment, using 10 new seeds per condition and comparing against the original RbRL settings with paired bootstrap confidence intervals. If the Cheetah improvement over original RbRL is not positive with non-overlapping intervals, the reported gains are selection artifacts rather than genuine optimization. A stronger version would hold out an environment not used at all, such as Hopper or Ant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result is that the optimized configuration (k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, LR 0.0005) gives better and more consistent performance than the original RbRL. For this claim to be supported, the chosen hyperparameters must be shown to perform well on data not used to select them. Instead, the 'Optimized RbRL' section selects hyperparameters as the 'best performing and most consistent' across Walker, Quadruped, and Cheetah, and then reports Figures 8 and 9 on those same three environments. For example, ArcTan is adopted after Figure 7 shows a Cheetah n=2 improvement from roughly 130 to 300, and k=1 is adopted after Table 1 shows k=1 is best only 'in most cases', with several entries having overlapping standard errors. No task, rating class, or seed set is held out. This makes the central claim compatible with selection over noise: picking the best of several configurations on a small number of runs will tend to look like an improvement even when the underlying effect is zero. The paper's disclosure that it is work in progress and its future human-subject tests are honest, but they do not address this internal-validity problem. This concern is more fundamental than the synthetic-rater proxy discussed by the reader: if the tuned gains do not reproduce on fresh evaluations, the recommended settings are not established even for synthetic raters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies hyperparameter optimization for rating-based reinforcement learning (RbRL), a reward-learning method that infers rewards from human ratings. The authors evaluate eight optimization axes: reward boundary, confidence index k, a modified class-probability function (Eq. 5), optimizer (Adam vs. AdamW), number of hidden layers, dropout rate, activation function, and learning rate, on Walker, Quadruped, and Cheetah with synthetic raters. They then propose an 'optimized RbRL' configuration (k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, LR 0.0005) and report that it improves performance and consistency over the original RbRL in Figures 8 and 9. The paper is explicitly preliminary and lists human-subject tests and additional environments as future work.","tokens_in":9758,"tokens_out":6462,"duration_ms":61132,"significance":"If the reported gains reproduce on data not used to select the configuration, the paper would provide a useful empirical reference for RbRL, a relatively underexplored alternative to preference-based RL. The systematic sweep over eight hyperparameters, the proposal of a new class-probability function, and the transparent acknowledgment of the work-in-progress status are strengths. The main significance is currently limited by the lack of independent validation of the recommended configuration, which is the central issue addressed in the major comments.","major_comments":[{"comment":"The central claim that the optimized configuration 'can show better performance and consistency' is not supported because the configuration was selected as the 'best performing and most consistent' on the same three environments used for the comparison. This is a selection-on-the-evaluation-data design: choosing the best among several configurations by the same metric later used to report improvement will appear as a gain even under pure noise, especially with the small seed counts and overlapping standard errors in Tables 1-3. Please add a validation protocol in which the hyperparameters are chosen on one set of environments, rating classes, or seeds and evaluated on a disjoint held-out set, or at minimum use nested cross-validation and report full per-configuration seed distributions. This point is load-bearing for the manuscript's main contribution.","section":"Optimized RbRL, Figures 8 and 9"},{"comment":"The quantitative evidence for the recommended settings is fragile. Several entries report mean and standard error of identical magnitude (Table 1, Quadruped RbRL (n=6), k=0.1: 228.37 ± 228.37; Table 2, Quadruped RbRL (n=6), Eq. (5), 30% dropout: 103.21 ± 103.21), indicating single-run results or degenerate variation, and no significance tests or confidence intervals are provided. The statement that k=1 'can produce best performance in most cases' is therefore not established; for example, in Table 1, Walker RbRL (n=6) with Eq. (3) gives k=1 = 931.09 ± 9.45 versus k=10 = 912.07 ± 1.02, which is within the reported variation. Please report the number of seeds for every condition, show the full seed-wise results, and add paired significance tests or effect sizes.","section":"Tables 1-3 and Confidence Index"},{"comment":"The conclusions that the best boundary is 25 for n=2 and 20 for n=6 with Eq. (3), and 15 for both with Eq. (5), are based on visual inspection of five runs without statistical comparison. The text uses 'suggests,' but the later choice of the optimized configuration implicitly relies on these differences. Please provide quantitative comparisons, such as means with error bars over seeds and paired tests across seed sets, and justify why equal partition of an arbitrary maximum is the right parameterization for the boundary sweep.","section":"Reward Boundary, Figure 2"},{"comment":"The practical recommendation of specific hyperparameters, especially k=1 and the reward boundaries, is conditioned on the synthetic-rating model described in the Optimization section, where 'synthetic ratings are generated based on the true environment reward.' A threshold on the true reward is not shown to be a faithful proxy for human raters, and the paper lists human-subject tests as future work. The claims about what 'users' should do in the Confidence Index section should be rephrased as claims about this synthetic-rater model, or supported by a transfer experiment.","section":"Optimization, Discussion"}],"minor_comments":[{"comment":"There are several typos, including 'well-know' in the Introduction, 'learing' in the Introduction, 'b to considered' in the Optimization section, and 'preforming' in the Optimized RbRL section; please proofread the manuscript.","section":"Throughout"},{"comment":"Lecun Tanh is not defined or cited; its functional form and reference should be specified in the text or caption.","section":"Activation Functions"},{"comment":"The reported Cheetah improvement 'from 130 to almost 300' is not traceable to a clearly labeled baseline in Figure 7; please state explicitly where the 130 value comes from.","section":"Activation Functions, Figure 7"},{"comment":"The caption of Figure 2 does not state the number of runs or whether the same initializations are used across boundary conditions; the text mentions five runs, but this should also appear in the caption.","section":"Reward Boundary, Figure 2"},{"comment":"The number of seeds is not stated for Table 1 or Table 3, although other sections mention five runs or three runs; please standardize seed reporting across all tables and figures.","section":"Tables 1 and 3"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test note that the selection-on-evaluation-data issue is the decisive one. Because the manuscript is explicitly work in progress and the missing held-out validation can in principle be added with additional experiments, I recommend major revision rather than rejection. The paper would benefit from a clear statement that the recommended hyperparameters are a preliminary proposal until independent validation is completed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing about this paper: the central claim does not hold. The authors tune hyperparameters on Walker, Quadruped, and Cheetah, then report the tuned configuration's performance on those same three environments. That is selection over noise. Picking the best of several configurations based on the exact runs you later report as evidence will look like an improvement even if the underlying effect is zero. The stress-test note is right, and it is more fundamental than the synthetic-rater concern.\n\nWhat the paper does well is modest but real. It extends RbRL with a new class probability function, Eq (5), which measures distance to the midpoint of a rating class instead of distance to the boundaries. That is a sensible variant to try, even if the results in Table 1 are inconsistent across tasks. The paper also surveys several RbRL-specific hyperparameters (reward boundary, confidence index, rating class count) alongside standard ML choices, and it is transparent about being work in progress. The writing is clear, and the authors do not oversell; the discussion of limitations is honest.\n\nThe soft spots are load-bearing. First, the circular comparison: the \"Optimized RbRL\" section picks k=1, AdamW, two hidden layers, 5% dropout, ArcTan, and LR 0.0005 based on the same environments shown in Figures 8 and 9. No task, rating class, or seed is held out. Second, the tables contain entries where mean and standard error are equal (e.g., Table 1, Quadruped n=6, k=0.1: 228.37 ± 228.37), which indicates single-run results. No significance tests are reported. Third, the \"over 100% improvement\" in Cheetah n=2 comes from selecting ArcTan after seeing Figure 7, which is the same selection problem in miniature.\n\nThe synthetic-ratings assumption is a real limitation, but it is not the main problem. Even for synthetic raters, the recommended settings are not established without a fresh evaluation. The paper says human subject tests are future work, which is fine, but it does not repair the internal-validity issue.\n\nWho is this paper for? Someone actively working on RbRL might find the hyperparameter survey a useful starting point for their own experiments, but they would need to rerun with a held-out protocol. I would not send this to a serious referee in its current form. The central claim is a selection artifact, and the fixing requires new experiments, not just editing.\n\nRecommendation: desk reject, but invite a resubmission if the authors add a held-out or cross-validated evaluation, report run counts for every entry, and release code.","headline":"An honest but methodologically flawed hyperparameter study: the headline improvement is manufactured by selecting hyperparameters on the same environments used for evaluation.","tokens_in":10282,"tokens_out":1777,"would_cite":false,"duration_ms":19803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hyperparameter tuning lifts rating-based RL performance by over 100%","keywords":["ratings-based reinforcement learning","hyperparameter optimization","reward learning from human feedback","class probability function","reward boundary","confidence index","AdamW optimizer","activation functions"],"falsifier":"Collect human ratings on segments from Walker, Quadruped, and Cheetah, train RbRL with the recommended settings and with the original defaults, and compare episodic returns; if the optimized settings do not beat the defaults under real-human ratings, the paper's central improvement claim does not transfer.","tokens_in":9289,"feed_emoji":"⚙️","tokens_out":6761,"duration_ms":55864,"temperature":0.7,"pith_summary":"This paper asks whether ratings-based reinforcement learning (RbRL), which learns a reward function from human ratings instead of environment rewards, can be made to work better by tuning its hyperparameters. The authors test eight optimization choices, three specific to RbRL (reward boundary, confidence index k, and a newly proposed class-probability function) and five standard machine-learning choices (optimizer, dropout, hidden-layer count, activation function, and learning rate). They report that a particular combination—k=1, AdamW, two hidden layers, 5% dropout, ArcTan activation, and a learning rate of 0.0005—delivers better and more consistent performance on Walker and Quadruped, and on Cheetah improves episodic return by over 100% for the two-class case. The paper is explicitly a work in progress, so the contribution is a set of provisional guidelines rather than a finished recipe.","feed_headline":"Hyperparameter tuning lifts rating-based RL performance by over 100%","feed_subtitle":"The paper's recommended settings—k=1, AdamW, two hidden layers, 5% dropout, ArcTan, and LR 0.0005—deliver the gains.","key_machinery":"The load-bearing object is the reward predictor in RbRL, a neural network trained by cross-entropy loss to classify segments into n rating classes. The predicted probability that a segment falls in class i is a softmax over a rating class; the original form, Equation (3), uses the product of distances from the segment's normalized cumulative reward to the lower and upper bounds of each class, scaled by a confidence index k. The paper's proposed alternative, Equation (5), replaces that product with the squared distance from the midpoint of the class bounds. The other machinery is the set of hyperparameters around that predictor: k sets how sharply the softmax discriminates, the reward boundary fixes where the class thresholds lie, and the standard choices (optimizer, dropout, hidden layers, activation, learning rate) control how well the network fits the rating signal.","core_discovery":"The central claim is that RbRL's performance and run-to-run consistency are substantially controlled by hyperparameters that the original method left at default values, and that a specific optimized configuration beats the defaults. On Cheetah with two rating classes, the optimized setup raises episodic reward from about 130 to nearly 300, an improvement of over 100%; on Walker and Quadruped the optimized configuration is better or comparable but more consistent across ten runs. The paper also proposes a new class-probability function, Equation (5), which measures the deviation of a segment's normalized cumulative reward from the midpoint of its rating class rather than the product of distances to the class bounds; this function is reported to be more stable across different reward boundary choices.","pith_inferences":["If synthetic threshold-based ratings prove to be a good stand-in for humans, the reward-boundary results imply that a stricter or looser rater changes the learned reward in predictable ways; matching the boundary to the rater's criteria could be a practical calibration step.","The stability of Equation (5) across reward boundaries suggests it may also be more robust to mislabeled segments, an error mode the paper lists as an open question; a natural test is to corrupt a fraction of synthetic ratings and compare Equations (3) and (5).","Because the recommended settings came from three continuous-control environments, they may not transfer directly to discrete or language tasks; the paper's open question about transfer to language tasks could be addressed by running the same ablation on a text environment."],"forward_implications":["Practitioners adopting RbRL can start from the recommended settings (k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, LR 0.0005) instead of the original defaults.","The proposed midpoint-based class probability function (Equation 5) is less sensitive to the choice of reward boundary, so it may be a safer default when a user's rating criteria are unknown.","With the optimized settings, RbRL's performance no longer degrades as the number of rating classes grows from 2 to 6; consistency across runs also improves.","The over-100% improvement on Cheetah n=2 shows that hyperparameter selection can be as important as algorithm choice in reward learning."],"supporting_citations":[{"why":"Introduces RbRL and the original class probability function (Equation 3) that this paper optimizes.","marker":"White et al. 2024"},{"why":"Introduces preference-based RL, the baseline approach RbRL extends and the paper compares against conceptually.","marker":"Christiano et al. 2017"},{"why":"Provides the BPref codebase that the experimental setup is built on.","marker":"Lee et al. 2021"},{"why":"Supplies the PPO algorithm used to train policies with the learned reward.","marker":"Schulman et al. 2017"},{"why":"Introduces the AdamW optimizer whose effect on RbRL is tested.","marker":"Loshchilov and Hutter 2019"},{"why":"Provides the dropout technique whose rate is ablated in this paper.","marker":"Srivastava et al. 2014"},{"why":"Benchmarks activation functions, motivating the ArcTan choice and the comparison set.","marker":"Dubey, Singh, and Chaudhuri 2022"},{"why":"Analyzes learning-rate policies, motivating the learning-rate sweep.","marker":"Wu et al. 2019"}],"fun_headline_variants":["Hyperparameter tuning more than doubles RbRL reward on Cheetah","New class-probability function for RbRL improves stability","RbRL reward jumps from 130 to 300 with tuned hyperparameters","Optimizing RbRL hyperparameters yields over 100% gain","Guidelines for tuning ratings-based RL boost Cheetah reward over 100%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole optimization study relies on ratings generated by thresholding the true environment reward as a stand-in for real human ratings; if real raters rate segments differently from those thresholds, the recommended settings—especially the reward boundary and confidence index k—may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Hyperparameter tuning more than doubles RbRL reward on Cheetah","New class-probability function for RbRL improves stability","RbRL reward jumps from 130 to 300 with tuned hyperparameters","Optimizing RbRL hyperparameters yields over 100% gain","Guidelines for tuning ratings-based RL boost Cheetah reward over 100%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3411,"prompt_tokens":841,"completion_tokens":2570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2478}},"tokens_in":457,"tokens_out":2570,"duration_ms":18563,"temperature":1.0,"reasoning_tokens":2478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:26.345421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human ratings on segments from Walker, Quadruped, and Cheetah, train RbRL with the recommended settings and with the original defaults, and compare episodic returns; if the optimized settings do not beat the defaults under real-human ratings, the paper's central improvement claim does not transfer.","supporting_citations":[{"cited_title":"J.; Waytowich, N.; and Cao, Y","cited_arxiv_id":null,"evidence_quote":"Introduces RbRL and the original class probability function (Equation 3) that this paper optimizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dropout technique whose rate is ablated in this paper."}],"review_version":1}