Pith. sign in

REVIEW 4 major objections 6 minor 59 references

Robust Reward Alignment via Hypothesis Space Batch Cutting

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By voting on each batch of preferences, an agent can keep the true reward inside its hypothesis space even when some labels are wrong, and with clean labels the method PAC-learns the reward in a bounded number of queries.

desk verdict A useful practical method for noisy preference learning, but the headline PAC guarantee does not apply to the implemented query process and the robustness claim leans on an oracle-tuned gamma. read the letter →

arxiv 2502.02921 v3 pith:5GJC53II submitted 2025-02-05 cs.LG

Zhixian Xie , Haode Zhang , Yizhe Feng , Wanxin Jin This is my paper · ORCID
classification cs.LG
keywords preference-basedreinforcementlearningrewardalignmenthypothesisspacecuttingvotingfunctionnoisyhumanfeedbackactivedisagreement-basedquerymodelpredictivecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a preference-based reward learner can tolerate a bounded fraction of wrong human labels without ever needing to identify which labels are wrong. The mechanism is hypothesis-space batch cutting: each batch of preferences defines a voting function over candidate rewards, and the learner keeps only rewards that receive enough votes, where the vote threshold is set by a conservativeness level $\gamma$. The paper proves that if each batch contains at most $\lceil \gamma N \rceil$ false labels, the true reward stays in every hypothesis space, so downstream planning and control remain aligned. For the all-true case it gives a probably approximately correct guarantee with a bounded query count. This matters because human feedback in real settings is noisy, and avoiding an explicit false-label filtering step keeps the learning procedure simple and geometrically interpretable.

What carries the argument

The load-bearing object is the voting function $V_i(\theta) = \sum_{j=1}^{N} H\bigl(f(\theta, \xi^0_{i,j}, \xi^1_{i,j}, y_{i,j})\bigr)$, where $H$ is the Heaviside step function and $f$ records whether reward parameter $\theta$ agrees with the preference label $y_{i,j}$ on a trajectory pair. It turns a batch of preferences into a geometric cut through the thresholded indicator $\mathbf{1}_{C_i}(\theta) = H\bigl(V_i(\theta) - \lfloor(1-\gamma)N\rfloor + 0.5\bigr)$. The threshold is chosen so that the true reward, which receives one vote from every true label, is preserved exactly when false labels are budgeted by $\gamma$. In the analysis, the disagreement coefficient and the VC-dimension of the reward model convert disagreement-based active sampling into a label-complexity bound.

What would settle it

Construct a single batch with $\lceil \gamma N \rceil + 1$ false labels arranged against the true reward: then $V_i(\theta_H) \le \lfloor (1-\gamma)N \rfloor - 1$, which is below the threshold $\lfloor (1-\gamma)N \rfloor - 0.5$, so $\theta_H$ is removed from the hypothesis space and the claimed invariance fails exactly beyond its stated budget.

Watch

Extended reading notes

Core claim

The central discovery is that the strict intersection cut of a preference batch, which fails on a single wrong label, can be replaced by a thresholded vote $C_i = \{\theta \mid V_i(\theta) > \lfloor (1-\gamma)N \rfloor - 0.5\}$, where $V_i(\theta)$ counts how many labels in the batch agree that $\theta$ is consistent with them. Because true labels always vote for $\theta_H$ and false labels vote against it, $\theta_H$ collects at least $\lfloor (1-\gamma)N \rfloor$ votes whenever the batch has at most $\lceil \gamma N \rceil$ false labels; the threshold is set just below that number, so $\theta_H$ is never cut out. With all-true preferences, $\gamma=0$ reduces the thresholded cut to the intersection, and disagreement-based queries give the PAC bound $K = O\bigl(\zeta(d\log\zeta + \log(\log(1/\epsilon)/\delta))\log(1/\epsilon)\bigr)$ with error at most $\epsilon$ and confidence at least $1-\delta$. The paper further reports that in six simulated control and manipulation tasks the method matches a standard preference-learning baseline at zero false rate and outperforms it and several robust variants at 10% to 30% false rates.

Load-bearing premise

The load-bearing premise is that every batch contains no more than $\lceil \gamma N \rceil$ false labels and that trajectory pairs are drawn independently from one fixed distribution; the implemented online algorithm generates its own trajectories from the evolving reward, so the fixed-distribution assumption is not met.

Editorial extensions

If this is right

  • With error-free preferences, HSBC learns a reward whose preference-prediction error is at most $\epsilon$ with probability at least $1-\delta$ using $K = O(\zeta(d\log\zeta + \log(\log(1/\epsilon)/\delta))\log(1/\epsilon))$ queries.
  • When every batch honors at most $\lceil \gamma N \rceil$ false labels, the true reward hypothesis is never removed, so planning stays aligned even at the 20% and 30% label-error rates reported in the experiments.
  • The method needs no separate classifier, no label-flipping step, and no assumption about the distribution of false preferences; robustness comes from the vote threshold alone.
  • Setting $\gamma=0$ recovers exact intersection cutting, so the robust procedure is a strict generalization of the clean-label version.
  • In the reported experiments, performance degrades only mildly from 0% to 30% false labels while a standard preference-learning baseline collapses, and the method also learns from real human volunteers with a conservativeness level of 40%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The PAC theorem assumes trajectory pairs are drawn independently from a fixed distribution $P_{XY}$, but the implemented agent generates trajectories from its own evolving model-predictive controller under the current reward ensemble; the paper does not close this gap, so the stated bound likely does not directly govern the online experiments. A concrete test would run HSBC on a fixed offline pref
  • The vote threshold implies a robustness-conservativeness trade-off: setting $\gamma$ above the true false rate preserves $\theta_H$ but cuts less volume per batch. The paper notes that a higher $\gamma$ increases query complexity, and one could quantify this by measuring hypothesis-space volume removed per query.
  • A reported weakness concerns a stochastic teacher that labels nearly tied trajectories inconsistently; this suggests that label noise concentrated inside the disagreement region can exceed a batch-level budget. An adaptive $\gamma$ or a disagreement-weighted voting scheme is a testable remedy the paper leaves implicit.
  • The geometric picture aligns with standard active learning, where the disagreement coefficient controls label complexity; HSBC can therefore be read as a label-noise-tolerant form of disagreement-based active learning rather than an entirely new statistical model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Hypothesis Space Batch Cutting (HSBC), an iterative reward-alignment method that maintains a hypothesis space of reward parameters and cuts it using batches of preference labels. For clean labels, it claims a PAC bound with active query complexity K = O(ζ(d log ζ + log(log(1/ε)/δ)) log(1/ε)) (Theorem 4.2). For noisy labels, it replaces the strict intersection cut with a thresholded vote Ci = {θ | Vi(θ) > ⌊(1−γ)N⌋ − 0.5} and proves that the true reward remains in the hypothesis space whenever each batch contains at most ⌈γN⌉ false labels (Lemma 5.2). Experiments in an MPC setting across six tasks compare HSBC with PEBBLE, RIME, SURF, MAE, and t-CE, showing competitive clean-label performance and improved robustness at 10–30% false-preference rates, plus ablations and real-human-data tests.

Significance. If the theoretical claims were established for the implemented algorithm, the paper would make a useful contribution: it provides a geometrically interpretable, query-efficient active-learning framework for preference-based reward learning, a conservative voting cut with a distribution-free containment guarantee under a false-label budget, and a broad empirical comparison including real human feedback. The code and website are advertised, which is helpful for reproducibility. However, the central PAC certificate is proved for a disagreement-based active-learning protocol that differs from the implemented MPC-based query process, and the robust guarantee is only a containment result, not a convergence or error bound. These gaps are load-bearing for the paper's headline claims, so the contribution is not yet fully established.

major comments (4)
  1. [§6.3, Algorithm 1; Appendix E.4, Eqs. (43)–(49)] Theorem 4.2's proof requires each batch Bi to consist of independent draws from P_XY conditioned on the disagreement region DIS(Vi) of the current version space. In the implemented HSBC (Algorithm 1, Section 6.3), trajectory pairs are produced by sampling-based MPC under the current reward ensemble (Eq. 21), with exploration noise and a disagreement filter (Eq. 22). This query process is neither i.i.d. nor drawn from a fixed P_XY: it depends on the current ensemble Ei, which changes with i and with the policy induced by the learned reward. Consequently, conditional on passing the disagreement filter, the queried pairs are not samples from P_X|DIS(Vi), so the conditional-error bound in Eq. (43) has no basis and the halving inequality in Eq. (49) is not established for the actual algorithm. Appendix E.4 does not close this gap. The advertised query-complexity certificate is a central contribution, and all experiments use the online MPC protocol, so this mismatch needs to be resolved either by changing the analysis to fit the protocol or by changing the protocol to satisfy the theorem's assumptions.
  2. [§7, Fig. 5 caption; §5.1, Lemma 5.2] The robust guarantee of Lemma 5.2 is conditional on a per-batch false-label budget of at most ⌈γN⌉. In the simulated-human experiments, the conservativeness level is set equal to the actual false rate (Fig. 5 caption), and false labels are generated by 'a random selection of human preference labels is flipped' (Section 7). For N = 10 and a 30% false rate, the probability that a batch contains more than 3 false labels is about 0.35, so Lemma 5.2's condition is violated on a substantial fraction of batches even though the average rate matches γ. The experiments therefore validate robustness under a favorable calibration of γ, not under the unknown-false-preference scenario advertised in the abstract. Please either set γ as a true upper bound in the experiments, report the empirical maximum number of false labels per batch, or provide results with γ strictly above the unknown rate across all tasks.
  3. [Appendix E.4, Eq. (44)] The proof of Theorem 4.2 asserts log(N/d) ≤ log ζ 'with proper choice of c, c′'. From N = c ζ(d log ζ + log(1/δ′)), we have N/d ≈ c ζ log ζ, so log(N/d) ≤ log ζ would require c ζ log ζ ≤ ζ, i.e., c log ζ ≤ 1, which is false for ζ < e^{1/c}. Unless additional assumptions on ζ or a different choice of N are supplied, the step from Eq. (44) to the halving inequality Eq. (49) is not justified for all ζ > 0. This is load-bearing for the query-complexity bound, so the proof needs a corrected bound or an explicit restriction on the disagreement coefficient.
  4. [§5, Lemma 5.2; §3, Q2] The robust section establishes only that the true reward θH remains in every hypothesis space when the false-label budget holds; it does not provide any PAC-style guarantee that the returned ensemble has low preference-prediction error err(rθ) under false labels. The problem statement Q2 asks whether the agent can 'still learn θH regardless of false preferences', but the theory never bounds err(rθ) for the robust procedure, nor does it show that the hypothesis space shrinks to a small error region in the presence of false labels. The empirical results suggest graceful degradation, but the 'provable robustness' claim in the abstract is stronger than the proven containment result. Please state this limitation explicitly or add an error bound for the robust algorithm.
minor comments (6)
  1. [Algorithm 1] Line 3 contains a typo: 'Sample an assemble Ei' should read 'Sample an ensemble Ei'.
  2. [Eq. (22)] The disagreement score formula is written as '4n+Ei n−Ei /N^2', which is ambiguous; it should be typeset as 4 n+_{Ei} n−_{Ei} / N^2, with n+_{Ei} and n−_{Ei} defined explicitly.
  3. [Appendix E.3] The passive PAC bound uses the symbol 'LP ASS' and later switches to 'M' and 'N' without defining the relationship; please use the batch size N consistently throughout the proof.
  4. [Section 7.4, Table 3] In the Walker row, the correlation at 0% false rate (0.584±0.035) is lower than at 10% (0.636±0.060), which is inconsistent with the text's claim of strong correlations across all rates; this deserves a comment or a corrected interpretation.
  5. [References] The Uni-RLHF reference 'Yuan et al.' lacks a year, venue, and page/arXiv information; please complete it.
  6. [Fig. 5 caption, Section 7.1] The caption states that the conservativeness level equals the actual false rate, but Section 5.1 frames γ as an unknown-rate hyperparameter; please reconcile this in the main text and discuss the limitation that this favorable calibration does not test unknown false rates.

Circularity Check

1 steps flagged · score 4.0 of 10

Lemma 5.2's robustness guarantee restates the gamma false-label budget as a vote threshold; Theorem 4.2 is an independent active-learning bound.

  1. self definitional [Section 5.1, Eq. (16); Lemma 5.2; Appendix D.3]
    "With the conservativeness level γ, one can change (9) into Ci = {θ|Vi(θ) > ⌊(1 − γ)N ⌋ −0.5}, ... Lemma 5.2. With the conservativeness level γ and replacing (9) with Ci in (16), the HSBC Algorithm will have θH ∈ Ci and θH ∈ Θi, i=1,2,3...I, regardless of false human preference."

    The cut threshold in (16) is set exactly one half-vote above ⌊(1−γ)N⌋, the vote count that θH is guaranteed to receive whenever a batch contains at most ⌈γN⌉ false labels. The proof of Lemma 5.2 (Appendix D.3) only counts these guaranteed correct votes and then invokes (16) to conclude θH ∈ Ci. Thus the robustness claim is the false-label budget rewritten as the threshold; it is true by construction of (16), not an independent property established about the learning process.

full rationale

The paper's principal PAC result (Theorem 4.2, Appendix E) is not circular: it gives a self-contained disagreement-based active-learning argument under an explicit i.i.d. P_XY assumption and a finite VC-dimension, with no load-bearing reliance on the authors' prior work. The cited self-work (Xie et al., 2024) is contextual, not load-bearing. The one definition-level reduction is Lemma 5.2: once γ is assumed to upper-bound the false-preference count per batch, the chosen threshold in (16) is precisely the vote level that keeps θH, so the 'provable robustness' follows immediately from the assumption and the cut definition. This is a partial, by-construction circularity in the robustness guarantee, but it does not invalidate the independent query-complexity analysis. A separate correctness concern—Theorem 4.2 assumes queries are i.i.d. from a fixed P_XY while Algorithm 1 generates trajectories from a shifting MPC reward ensemble—is a validity gap rather than a circularity and is not scored here. Experiments also set γ equal to the actual false rate, which supplies the guarantee's input but is an experimental-design choice.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theoretical results rest on four assumptions: an in-space true reward, deterministic preference rationality, a per-batch false-label budget gamma N, and an i.i.d. fixed distribution over trajectory pairs. The last is not satisfied by the implemented MPC-based querying; the third is an unverified hyperparameter in practice.

free parameters (1)
  • conservativeness level gamma = 0, 0.1, 0.2, 0.3 (equal to simulated false rate in Fig. 5); 0.4 for real human data
    gamma is the assumed upper bound on the fraction of false preferences per batch. Its value is chosen per experiment to match or exceed the true false rate; the method's robustness guarantee holds only if gamma is a valid bound. In Fig. 5 it is set equal to the actual false rate, so the experiments do not evaluate robustness under an unknown rate.
assumptions (5)
  • domain assumption True reward theta_H exists in the initial hypothesis space Theta_0 = R^r.
    Section 3 and Section 6.1; all guarantees rely on theta_H being in Theta_0.
  • domain assumption Human preferences follow the threshold rationality model in Eq. (2), so true labels are deterministic comparisons of J_theta_H.
    Section 3; if humans are stochastic or use a different model, Lemmas 4.1 and 5.2 need not hold.
  • ad hoc to paper Each preference batch contains at most ceil(gamma N) false labels.
    Section 5.1; Lemma 5.2 is a direct consequence of this assumption, and the paper provides no mechanism to certify gamma in practice.
  • domain assumption Trajectory pairs are drawn i.i.d. from a fixed distribution P_XY over pairs, and the version-space disagreement region is accessible for sampling.
    Appendix E.3; required for the PAC bound but contradicted by Section 6.3's MPC-generated rollouts.
  • standard math The concept class of reward-induced classifiers has finite VC dimension d and a finite disagreement coefficient zeta.
    Appendix E.2, used in Theorem 4.2; standard active learning conditions, not verified for the neural reward class used in experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust Reward Alignment via Hypothesis Space Batch Cutting." pith.science (2026). https://pith.science/paper/5GJC53II

@misc{pith2026250202921,
  author       = {Pith},
  title        = {Pith review of: Robust Reward Alignment via Hypothesis Space Batch Cutting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GJC53II}},
  note         = {Machine review of arXiv:2502.02921}
}
read the original abstract

Reward design in reinforcement learning and optimal control is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in a model predictive control setting across diverse tasks. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences.

Figures

Figures reproduced from arXiv: 2502.02921 by the authors.

Figure 1
Figure 1. Illustration of update from Θi to Θi+1. Three constraints are induced from a preference batch of size 3, with the simplified notation fi,j (θ) = f(θ, ξ 0 i,j , ξ 1 i,j , yi,j ). Red arrows are the di￾rections of constraints, i.e., the region of {θ|fi,j (θ) ≥ 0}. New Θi+1 is the regions in Θi satisfying all constraints. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of disagreement-based preference query. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Geometry Interpretation of the robust HSBC with false [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Task environments of experiments. 3 dm-control tasks: including Cartpole-Swingup, Walker￾Walk, Humanoid-Standup. We directly use Cartpole, Walker and Humanoid to denote the tasks. 2 in-hand dexterous manipulation tasks: An Allegro hand in-hand reorients a cube and bunn…
Figure 5
Figure 5. Figure 5: Learning curves for different tasks (rows) under different rates (columns) of false human preferences. The conservativeness level [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation study in Walker-Walk task for the choices of (a): conservativeness level [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Reward learning with real human preference. HSBC [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the Go2-Standup task, the goal is to reach a stand-up posture with two feet. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Learning curves of the methods used for comparison. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 37 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    April: Active preference learning-based reinforcement learning

    Akrour, R., Schoenauer, M., and Sebag, M. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp.\ 116--131. Springer, 2012

    2012

  3. [3]

    Programming by feedback

    Akrour, R., Schoenauer, M., Sebag, M., and Souplet, J.-C. Programming by feedback. In International Conference on Machine Learning, volume 32, pp.\ 1503--1511. JMLR. org, 2014

    work page 2014

  4. [4]

    K., Anil, R., and Koren, T

    Amid, E., Warmuth, M. K., Anil, R., and Koren, T. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  5. [5]

    Fine-tuning language models to find agreement among humans with diverse preferences

    Bakker, M., Chadwick, M., Sheahan, H., Tessler, M., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M., et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35: 0 38176--38189, 2022

    2022

  6. [6]

    L., Harvey, N., Liaw, C., and Mehrabian, A

    Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20 0 (63): 0 1--17, 2019

    2019

  7. [7]

    and Sadigh, D

    Biyik, E. and Sadigh, D. Batch active preference-based learning of reward functions. In Conference on robot learning, pp.\ 519--528. PMLR, 2018

    work page 2018

  8. [8]

    Batch active learning of reward functions from human preferences

    Biyik, E., Anari, N., and Sadigh, D. Batch active learning of reward functions from human preferences. ACM Transactions on Human-Robot Interaction, 13 0 (2): 0 1--27, 2024

    work page 2024

Show all 59 references
  1. [9]

    J., and Sadigh, D

    B y k, E., Huynh, N., Kochenderfer, M. J., and Sadigh, D. Active preference-based gaussian process regression for reward learning and optimization. The International Journal of Robotics Research, 43 0 (5): 0 665--684, 2024

    2024

  2. [10]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  3. [11]

    K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T

    Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T. T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P. J., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., ...

    2023

  4. [12]

    Rime: Robust preference-based reinforcement learning with noisy preferences

    Cheng, J., Xiong, G., Dai, X., Miao, Q., Lv, Y., and Wang, F.-Y. Rime: Robust preference-based reinforcement learning with noisy preferences. In International Conference on Machine Learning, pp.\ 8229--8247. PMLR, 2024

    2024

  5. [13]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    Active reward learning

    Daniel, C., Viering, M., Metz, J., Kroemer, O., and Peters, J. Active reward learning. In Robotics: Science and systems, volume 98, 2014

    2014

  7. [15]

    Feng, L., Shu, S., Lin, Z., Lv, F., Li, L., and An, B. Can cross entropy loss be robust to label noise? In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp.\ 2206--2212, 2021

    2021

  8. [16]

    Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  9. [17]

    Dextreme: Transfer of agile in-hand manipulation from simulation to reality

    Handa, A., Allshire, A., Makoviychuk, V., Petrenko, A., Singh, R., Liu, J., Makoviichuk, D., Van Wyk, K., Zhurkevich, A., Sundaralingam, B., et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics an...

    2023

  10. [18]

    A bound on the label complexity of agnostic active learning

    Hanneke, S. A bound on the label complexity of agnostic active learning. In Proceedings of the 24th international conference on Machine learning, pp.\ 353--360, 2007

    2007

  11. [19]

    Contrastive preference learning: Learning from human feedback without rl

    Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W., and Sadigh, D. Contrastive preference learning: Learning from human feedback without rl. In International Conference on Learning Representations, 2024

    2024

  12. [20]

    Hejna III, D. J. and Sadigh, D. Few-shot preference learning for human-in-the-loop rl. In Conference on Robot Learning, pp.\ 2014--2025. PMLR, 2023

    2014

  13. [21]

    J., Kim, J., Kwak, M

    Heo, J., Lee, Y. J., Kim, J., Kwak, M. G., Park, Y. J., and Kim, S. B. Mixing corrupted preferences for robust and feedback-efficient preference-based reinforcement learning. Knowledge-Based Systems, 309: 0 112824, 2025

    2025

  14. [22]

    Anymal parkour: Learning agile navigation for quadrupedal robots

    Hoeller, D., Rudin, N., Sako, D., and Hutter, M. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9 0 (88): 0 eadi7566, 2024

    2024

  15. [23]

    Bayesian active learning for classification and preference learning

    Houlsby, N., Husz \'a r, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. stat, 1050: 0 24, 2011

    2011

  16. [24]

    Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo

    Howell, T., Gileadi, N., Tunyasuvunakool, S., Zakka, K., Erez, T., and Tassa, Y. Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo . dec 2022. doi:10.48550/arXiv.2212.00541. URL https://arxiv.org/abs/2212.00541

    2022 doi arXiv

  17. [25]

    Reward learning from human preferences and demonstrations in atari

    Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018

    2018

  18. [26]

    Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels

    Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp.\ 2304--2313. PMLR, 2018

    2018

  19. [27]

    D., Lu, Z., and Mou, S

    Jin, W., Murphey, T. D., Lu, Z., and Mou, S. Learning from human directional corrections. IEEE Transactions on Robotics, 39 0 (1): 0 625--644, 2022

    2022

  20. [28]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  21. [29]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  22. [30]

    Robust inference via generative classifiers for handling noisy labels

    Lee, K., Yun, S., Lee, K., Lee, H., Li, B., and Shin, J. Robust inference via generative classifiers for handling noisy labels. In International conference on machine learning, pp.\ 3763--3772. PMLR, 2019

    2019

  23. [31]

    B-pref: Benchmarking preference-based reinforcement learning

    Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref: Benchmarking preference-based reinforcement learning. In 35th Conference on Neural Information Processing Systems (NeurIPS). Neural Information Processing Systems Foundation, 2021 a

    2021

  24. [32]

    M., and Abbeel, P

    Lee, K., Smith, L. M., and Abbeel, P. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of P...

    2021

  25. [33]

    Li, Y., Das, S., and Taylor, M. E. Candere-coach: Reinforcement learning from noisy feedback. arXiv preprint arXiv:2409.15521, 2024

    2024 arXiv

  26. [34]

    Reward uncertainty for exploration in preference-based reinforcement learning

    Liang, X., Shu, K., Lee, K., and Abbeel, P. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC

    2022

  27. [35]

    Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning

    Liu, R., Bai, F., Du, Y., and Yang, Y. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 22270--22284, 2022

    2022

  28. [36]

    Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp.\ 6448--6458

    Lukasik, M., Bhojanapalli, S., Menon, A., and Kumar, S. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp.\ 6448--6458. PMLR, 2020

    2020

  29. [37]

    Normalized loss functions for deep learning with noisy labels

    Ma, X., Huang, H., Wang, Y., Romano, S., Erfani, S., and Bailey, J. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pp.\ 6543--6553. PMLR, 2020

    2020

  30. [38]

    Learning multimodal rewards from rankings

    Myers, V., Biyik, E., Anari, N., and Sadigh, D. Learning multimodal rewards from rankings. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp.\ 342--352. PMLR, 08--11 Nov 2022

    2022

  31. [39]

    Active reward learning from online preferences

    Myers, V., Bıyık, E., and Sadigh, D. Active reward learning from online preferences. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 7511--7518, 2023. doi:10.1109/ICRA48891.2023.10160439

    2023

  32. [40]

    Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning

    Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203.10050, 2022

    2022 arXiv

  33. [41]

    In-hand object rotation via rapid motor adaptation

    Qi, H., Kumar, A., Calandra, R., Ma, Y., and Malik, J. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pp.\ 1722--1732. PMLR, 2023

    2023

  34. [42]

    Real-world humanoid locomotion with reinforcement learning

    Radosavovic, I., Xiao, T., Zhang, B., Darrell, T., Malik, J., and Sreenath, K. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9 0 (89): 0 eadi9579, 2024

    2024

  35. [43]

    D., Sastry, S

    Sadigh, D., Dragan, A. D., Sastry, S. S., and Seshia, S. A. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017. URL https://api.semanticscholar.org/CorpusID:12226563

    2017

  36. [44]

    Active Learning

    Settles, B. Active Learning. Morgan & Claypool Publishers, 2012. ISBN 1608457257

    2012

  37. [45]

    and Joachims, T

    Shivaswamy, P. and Joachims, T. Coactive learning. Journal of Artificial Intelligence Research, 53: 0 1--40, 2015

    2015

  38. [46]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  39. [47]

    Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning

    Tsounis, V., Alge, M., Lee, J., Farshidian, F., and Hutter, M. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, 5 0 (2): 0 3699--3706, 2020

    2020

  40. [48]

    Statistical learning theory

    Vapnik, V. Statistical learning theory. John Wiley & Sons google schola, 2: 0 831--842, 1998

    1998

  41. [49]

    Model predictive path integral control using covariance variable importance sampling

    Williams, G., Aldrich, A., and Theodorou, E. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01149, 2015

    2015 arXiv

  42. [50]

    A survey of preference-based reinforcement learning methods

    Wirth, C., Akrour, R., Neumann, G., and F \"u rnkranz, J. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 0 (136): 0 1--46, 2017

    2017

  43. [51]

    J., and Jin, W

    Xie, Z., Zhang, W., Ren, Y., Wang, Z., Pappas, G. J., and Jin, W. Safe mpc alignment with human directional feedback. arXiv preprint arXiv:2407.04216, 2024

    2024

  44. [52]

    Reinforcement learning from diverse human preferences

    Xue, W., An, B., Yan, S., and Xu, Z. Reinforcement learning from diverse human preferences. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24, 2024. ISBN 978-1-956792-04-1. doi:10.24963/ijcai.2024/586. URL https://doi.org/1...

    2024 doi

  45. [53]

    W., Zhang, Y., Sun, J., Zhang, C., and Zhang, R

    Yao, J., Wang, J., Tsang, I. W., Zhang, Y., Sun, J., Zhang, C., and Zhang, R. Deep learning from noisy image labels with quality embedding. IEEE Transactions on Image Processing, 28 0 (4): 0 1909--1922, 2018

    1909

  46. [54]

    Rotating without seeing: Towards in-hand dexterity through touch

    Yin, Z.-H., Huang, B., Qin, Y., Chen, Q., and Wang, X. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023

    2023

  47. [55]

    Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback

    Yuan, Y., Jianye, H., Ma, Y., Dong, Z., Liang, H., Liu, J., Feng, Z., Zhao, K., and Zheng, Y. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In The Twelfth International Conference on Learning Representations

  48. [56]

    N., and Lopez-Paz, D

    Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1Ddp1-Rb

    2018

  49. [57]

    Robust curriculum learning: from clean label detection to noisy label self-correction

    Zhou, T., Wang, S., and Bilmes, J. Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations, 2020

    2020

  50. [58]

    A., Atkeson, C

    Zucker, M., Bagnell, J. A., Atkeson, C. G., and Kuffner, J. An optimization approach to rough terrain locomotion. In 2010 IEEE International Conference on Robotics and Automation, pp.\ 3589--3595. IEEE, 2010

    2010

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.