Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Performance Optimization of Ratings-Based Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hyperparameter tuning lifts rating-based RL performance by over 100%

desk verdict An honest but methodologically flawed hyperparameter study: the headline improvement is manufactured by selecting hyperparameters on the same environments used for evaluation. read the letter →

arxiv 2501.07755 v1 pith:MQIHFG5Z submitted 2025-01-13 cs.LG cs.AI

Evelyn Rose , Devin White , Mingkang Wu , Vernon Lawhern , Nicholas R. Waytowich , Yongcan Cao This is my paper · ORCID
classification cs.LGcs.AI
keywords ratings-basedreinforcementlearninghyperparameteroptimizationrewardfromhumanfeedbackclassprobabilityfunctionboundaryconfidenceindexAdamWoptimizeractivationfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether ratings-based reinforcement learning (RbRL), which learns a reward function from human ratings instead of environment rewards, can be made to work better by tuning its hyperparameters. The authors test eight optimization choices, three specific to RbRL (reward boundary, confidence index k, and a newly proposed class-probability function) and five standard machine-learning choices (optimizer, dropout, hidden-layer count, activation function, and learning rate). They report that a particular combination—k=1, AdamW, two hidden layers, 5% dropout, ArcTan activation, and a learning rate of 0.0005—delivers better and more consistent performance on Walker and Quadruped, and on Cheetah improves episodic return by over 100% for the two-class case. The paper is explicitly a work in progress, so the contribution is a set of provisional guidelines rather than a finished recipe.

What carries the argument

The load-bearing object is the reward predictor in RbRL, a neural network trained by cross-entropy loss to classify segments into n rating classes. The predicted probability that a segment falls in class i is a softmax over a rating class; the original form, Equation (3), uses the product of distances from the segment's normalized cumulative reward to the lower and upper bounds of each class, scaled by a confidence index k. The paper's proposed alternative, Equation (5), replaces that product with the squared distance from the midpoint of the class bounds. The other machinery is the set of hyperparameters around that predictor: k sets how sharply the softmax discriminates, the reward boundary fixes where the class thresholds lie, and the standard choices (optimizer, dropout, hidden layers, activation, learning rate) control how well the network fits the rating signal.

What would settle it

Collect human ratings on segments from Walker, Quadruped, and Cheetah, train RbRL with the recommended settings and with the original defaults, and compare episodic returns; if the optimized settings do not beat the defaults under real-human ratings, the paper's central improvement claim does not transfer.

Watch

Extended reading notes

Core claim

The central claim is that RbRL's performance and run-to-run consistency are substantially controlled by hyperparameters that the original method left at default values, and that a specific optimized configuration beats the defaults. On Cheetah with two rating classes, the optimized setup raises episodic reward from about 130 to nearly 300, an improvement of over 100%; on Walker and Quadruped the optimized configuration is better or comparable but more consistent across ten runs. The paper also proposes a new class-probability function, Equation (5), which measures the deviation of a segment's normalized cumulative reward from the midpoint of its rating class rather than the product of distances to the class bounds; this function is reported to be more stable across different reward boundary choices.

Load-bearing premise

The whole optimization study relies on ratings generated by thresholding the true environment reward as a stand-in for real human ratings; if real raters rate segments differently from those thresholds, the recommended settings—especially the reward boundary and confidence index k—may not transfer.

Editorial extensions

If this is right

  • Practitioners adopting RbRL can start from the recommended settings (k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, LR 0.0005) instead of the original defaults.
  • The proposed midpoint-based class probability function (Equation 5) is less sensitive to the choice of reward boundary, so it may be a safer default when a user's rating criteria are unknown.
  • With the optimized settings, RbRL's performance no longer degrades as the number of rating classes grows from 2 to 6; consistency across runs also improves.
  • The over-100% improvement on Cheetah n=2 shows that hyperparameter selection can be as important as algorithm choice in reward learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If synthetic threshold-based ratings prove to be a good stand-in for humans, the reward-boundary results imply that a stricter or looser rater changes the learned reward in predictable ways; matching the boundary to the rater's criteria could be a practical calibration step.
  • The stability of Equation (5) across reward boundaries suggests it may also be more robust to mislabeled segments, an error mode the paper lists as an open question; a natural test is to corrupt a fraction of synthetic ratings and compare Equations (3) and (5).
  • Because the recommended settings came from three continuous-control environments, they may not transfer directly to discrete or language tasks; the paper's open question about transfer to language tasks could be addressed by running the same ablation on a text environment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies hyperparameter optimization for rating-based reinforcement learning (RbRL), a reward-learning method that infers rewards from human ratings. The authors evaluate eight optimization axes: reward boundary, confidence index k, a modified class-probability function (Eq. 5), optimizer (Adam vs. AdamW), number of hidden layers, dropout rate, activation function, and learning rate, on Walker, Quadruped, and Cheetah with synthetic raters. They then propose an 'optimized RbRL' configuration (k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, LR 0.0005) and report that it improves performance and consistency over the original RbRL in Figures 8 and 9. The paper is explicitly preliminary and lists human-subject tests and additional environments as future work.

Significance. If the reported gains reproduce on data not used to select the configuration, the paper would provide a useful empirical reference for RbRL, a relatively underexplored alternative to preference-based RL. The systematic sweep over eight hyperparameters, the proposal of a new class-probability function, and the transparent acknowledgment of the work-in-progress status are strengths. The main significance is currently limited by the lack of independent validation of the recommended configuration, which is the central issue addressed in the major comments.

major comments (4)
  1. [Optimized RbRL, Figures 8 and 9] The central claim that the optimized configuration 'can show better performance and consistency' is not supported because the configuration was selected as the 'best performing and most consistent' on the same three environments used for the comparison. This is a selection-on-the-evaluation-data design: choosing the best among several configurations by the same metric later used to report improvement will appear as a gain even under pure noise, especially with the small seed counts and overlapping standard errors in Tables 1-3. Please add a validation protocol in which the hyperparameters are chosen on one set of environments, rating classes, or seeds and evaluated on a disjoint held-out set, or at minimum use nested cross-validation and report full per-configuration seed distributions. This point is load-bearing for the manuscript's main contribution.
  2. [Tables 1-3 and Confidence Index] The quantitative evidence for the recommended settings is fragile. Several entries report mean and standard error of identical magnitude (Table 1, Quadruped RbRL (n=6), k=0.1: 228.37 ± 228.37; Table 2, Quadruped RbRL (n=6), Eq. (5), 30% dropout: 103.21 ± 103.21), indicating single-run results or degenerate variation, and no significance tests or confidence intervals are provided. The statement that k=1 'can produce best performance in most cases' is therefore not established; for example, in Table 1, Walker RbRL (n=6) with Eq. (3) gives k=1 = 931.09 ± 9.45 versus k=10 = 912.07 ± 1.02, which is within the reported variation. Please report the number of seeds for every condition, show the full seed-wise results, and add paired significance tests or effect sizes.
  3. [Reward Boundary, Figure 2] The conclusions that the best boundary is 25 for n=2 and 20 for n=6 with Eq. (3), and 15 for both with Eq. (5), are based on visual inspection of five runs without statistical comparison. The text uses 'suggests,' but the later choice of the optimized configuration implicitly relies on these differences. Please provide quantitative comparisons, such as means with error bars over seeds and paired tests across seed sets, and justify why equal partition of an arbitrary maximum is the right parameterization for the boundary sweep.
  4. [Optimization, Discussion] The practical recommendation of specific hyperparameters, especially k=1 and the reward boundaries, is conditioned on the synthetic-rating model described in the Optimization section, where 'synthetic ratings are generated based on the true environment reward.' A threshold on the true reward is not shown to be a faithful proxy for human raters, and the paper lists human-subject tests as future work. The claims about what 'users' should do in the Confidence Index section should be rephrased as claims about this synthetic-rater model, or supported by a transfer experiment.
minor comments (5)
  1. [Throughout] There are several typos, including 'well-know' in the Introduction, 'learing' in the Introduction, 'b to considered' in the Optimization section, and 'preforming' in the Optimized RbRL section; please proofread the manuscript.
  2. [Activation Functions] Lecun Tanh is not defined or cited; its functional form and reference should be specified in the text or caption.
  3. [Activation Functions, Figure 7] The reported Cheetah improvement 'from 130 to almost 300' is not traceable to a clearly labeled baseline in Figure 7; please state explicitly where the 130 value comes from.
  4. [Reward Boundary, Figure 2] The caption of Figure 2 does not state the number of runs or whether the same initializations are used across boundary conditions; the text mentions five runs, but this should also appear in the caption.
  5. [Tables 1 and 3] The number of seeds is not stated for Table 1 or Table 3, although other sections mention five runs or three runs; please standardize seed reporting across all tables and figures.

Circularity Check

3 steps flagged · score 6.0 of 10

Optimized RbRL's reported improvement is a within-sample selection result: the hyperparameters are chosen as best on the same environments later used to claim better performance, with no held-out evaluation.

  1. fitted input called prediction [Optimized RbRL, paragraph 1 (selection sentence)]
    "We select the hyperparameters based on the best preforming and most consistent results across the three different environments."

    The hyperparameters k=1, AdamW, 2 hidden layers, 5% dropout, ArcTan, and LR 0.0005 are chosen because they performed best on Walker, Quadruped, and Cheetah. Those are exactly the environments used in the later Figure 8 and Figure 9 comparisons, so the optimized configuration is a fitted value, not an independent predictor. The paper reports no held-out task, rating class, or seed set for the optimized configuration, so the subsequent improvement claim is not statistically independent of the selection criterion.

  2. fitted input called prediction [Optimized RbRL, paragraph 1 (conclusion sentence)]
    "Figure 8 illustrates the performance difference between optimized RbRL and RbRL for n = 2 in Walker, Quadruped and Cheetah. It can be observed that the optimized RbRL can show better performance and consistency in Walker and Quadruped and much better performance in Cheetah."

    The sentence is presented as an observed result of the optimized configuration, but the configuration was defined by selecting the best-performing and most consistent hyperparameters on these same three environments. The conclusion is therefore not an independent evaluation: the same data that determined the configuration is reused as its test set. No unseen environment, rating class, or seed set is held out, so the reported better performance is compatible with selection over noise rather than a verified improvement.

1 more flagged steps
  1. fitted input called prediction [Activation Functions section, paragraph 3]
    "However, the use of ArcTan shows the best consistency and performance across all number of rating classes and environments. This is shown for Cheetah in Figure 7, where we can see for n = 2 the performance increased from 130 to almost 300 showing an improvement of over 100%."

    ArcTan is adopted because it was the best activation function in the same Walker, Quadruped, and Cheetah comparisons, and the over-100% Cheetah improvement is then quoted from the same comparative experiment that motivated the choice. The improvement is the selected maximum over the tested activation functions, not a result obtained from a configuration chosen before seeing Cheetah data.

full rationale

The paper's central claim is that a hyperparameter configuration selected as best performing and most consistent across Walker, Quadruped, and Cheetah then shows better performance and consistency on those same environments. This is the fitted-input-called-prediction pattern: the configuration is fit to the same benchmark data used to report the improvement. No held-out environment, rating class, or seed set is used, so the reported gains are not statistically independent of the selection procedure. The circularity here is not the self-citation of White et al. (2024), which is a normal baseline reference and not load-bearing; nor is the synthetic-human assumption a circular step, since it is a stated limitation rather than an input that forces the result. Equation (5) is introduced as a new proposal and tested empirically; lacking a derivation is a rigor concern, not a circularity. The circularity is specifically in the Optimized RbRL evaluation, where the selection criterion and the success metric coincide. Because this affects the paper's headline improvement claim, the score is elevated to 6. The paper's own disclosure that it is a work in progress and its plan for future human-subject tests are honest but do not repair the within-sample selection issue.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is a set of hyperparameter choices, each effectively fit to the test environments and then evaluated on the same environments.

free parameters (7)
  • Confidence index k = 1
    Selected as best/consistent across Walker, Quadruped, Cheetah in Table 1; introduced in Eq (3)/Eq (5).
  • Learning rate = 0.0005
    Chosen as best in Walker Table 3.
  • Dropout rate = 5%
    Chosen from Table 2 as generally beneficial, no clear universal value.
  • Number of hidden layers = 2
    Chosen as best in Cheetah Figure 4.
  • Activation function = ArcTan
    Chosen as best consistency/performance in Walker, Cheetah, Quadruped Figures 5-7.
  • Optimizer = AdamW
    Chosen over Adam based on Figure 3.
  • Reward boundary = 25 (Eq 3, n=2), 20 (Eq 3, n=6), 15 (Eq 5)
    Only tested in Cheetah; not used in final optimized config but investigated.
assumptions (4)
  • domain assumption RbRL reward predictor and class-boundary inference from White et al. 2024 are correct and used as-is.
    The paper builds on the RbRL codebase and equations; no re-derivation.
  • domain assumption Synthetic ratings generated by thresholding true environment reward model human rating behavior.
    Section Optimization: ratings are generated from true reward; no human subjects validation.
  • domain assumption PPO with entropy-based exploration reward for the first 32,000 timesteps is an appropriate RL backbone.
    Results section setup.
  • domain assumption Batch normalization and cross-entropy minimization in Eqs (3)-(4) produce a usable reward predictor.
    Taken from White et al. 2024 without re-derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Optimization of Ratings-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/MQIHFG5Z

@misc{pith2026250107755,
  author       = {Pith},
  title        = {Pith review of: Performance Optimization of Ratings-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQIHFG5Z}},
  note         = {Machine review of arXiv:2501.07755}
}
read the original abstract

This paper explores multiple optimization methods to improve the performance of rating-based reinforcement learning (RbRL). RbRL, a method based on the idea of human ratings, has been developed to infer reward functions in reward-free environments for the subsequent policy learning via standard reinforcement learning, which requires the availability of reward functions. Specifically, RbRL minimizes the cross entropy loss that quantifies the differences between human ratings and estimated ratings derived from the inferred reward. Hence, a low loss means a high degree of consistency between human ratings and estimated ratings. Despite its simple form, RbRL has various hyperparameters and can be sensitive to various factors. Therefore, it is critical to provide comprehensive experiments to understand the impact of various hyperparameters on the performance of RbRL. This paper is a work in progress, providing users some general guidelines on how to select hyperparameters in RbRL.

Figures

Figures reproduced from arXiv: 2501.07755 by the authors.

Figure 1
Figure 1. A schematic illustration of rating-based reinforce [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Episodic reward in Cheetah using different reward [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Episodic reward using Adam and AdamW in Walker. can be observed that RbRL performance and consistency of the model is best when using 2 layers for both equation (3) and (5) . However, the use of 3 layers can achieve reasonable performance and consistency for both equation (3) and (5). We believe that the performance difference between using 2 layers and 3 layers is due to the model potentially overfit￾ting. Further … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Episodic reward in Cheetah using a different num [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 8
Figure 8. Figure 8: Episodic reward using optimized parameters com [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 5
Figure 5. Figure 5: Episodic reward in Walker using different activa [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Episodic reward in Quadruped using different ac [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Episodic reward using optimized parameters com [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Task Reward Learning from Human Ratings

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A multi-task loss mixing rating classification, regression, and learned uncertainty weights improves reward learning from ratings over a classification-only baseline in synthetic control tasks.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    P.; Piot, B.; Kapturowski, S.; Sprechmann, P.; Vitvitskyi, A.; Guo, D.; and Blundell, C

    Badia, A. P.; Piot, B.; Kapturowski, S.; Sprechmann, P.; Vitvitskyi, A.; Guo, D.; and Blundell, C. 2020. Agent57: Outperforming the Atari Human Benchmark. arXiv:2003.13350

    arXiv 2020

  2. [2]

    Choi, J.; and Kim, K.-E. 2011. MAP inference for B ayesian inverse reinforcement learning. Advances in Neural Information Processing Systems, 24

    work page 2011

  3. [3]

    F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D

    Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  4. [4]

    R.; Singh, S

    Dubey, S. R.; Singh, S. K.; and Chaudhuri, B. B. 2022. Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark. arXiv:2109.14545

    arXiv 2022

  5. [5]

    Finn, C.; Levine, S.; and Abbeel, P. 2016. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the International Conference on Machine Learning, 49--58

    work page 2016

  6. [6]

    Lee, K.; Smith, L.; Dragan, A.; and Abbeel, P. 2021. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026

    arXiv 2021

  7. [7]

    Levine, S.; Popovic, Z.; and Koltun, V. 2011. Nonlinear inverse reinforcement learning with G aussian processes. Advances in Neural Information Processing Systems, 24

    work page 2011

  8. [8]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101

    arXiv 2019

Show all 19 references
  1. [9]

    Y.; Russell, S

    Ng, A. Y.; Russell, S. J.; et al. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2

    2000

  2. [10]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347

    2017 arXiv

  3. [11]

    I.; and Kashif, F

    Shafi, I.; Ahmad, J.; Shah, S. I.; and Kashif, F. M. 2006. Impact of Varying Neurons and Hidden Layers in Neural Network Architecture for a Time Frequency Application. In 2006 IEEE International Multitopic Conference, 188--193

    2006

  4. [12]

    Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929--1958

    2014

  5. [13]

    Szot, A.; Zhang, A.; Batra, D.; Kira, Z.; and Meier, F. 2022. BC-IRL : Learning Generalizable Reward Functions from Demonstrations. In The Eleventh International Conference on Learning Representations

    2022

  6. [14]

    Tang, C.; Abbatematteo, B.; Hu, J.; Chandra, R.; Martín-Martín, R.; and Stone, P. 2024. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes. arXiv:2408.03539

    2024 arXiv

  7. [15]

    J.; Waytowich, N.; and Cao, Y

    White, D.; Wu, M.; Novoseller, E.; Lawhern, V. J.; Waytowich, N.; and Cao, Y. 2024. Rating-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 10207--10215

    2024

  8. [16]

    Wu, Y.; Liu, L.; Bae, J.; Chow, K.-H.; Iyengar, A.; Pu, C.; Wei, W.; Yu, L.; and Zhang, Q. 2019. Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks. arXiv:1908.06477

    2019 arXiv

  9. [17]

    D.; Maas, A.; Bagnell, J

    Ziebart, B. D.; Maas, A.; Bagnell, J. A.; and Dey, A. K. 2008. Maximum entropy inverse reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence

    2008

  10. [18]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  11. [19]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.