Pith. sign in

REVIEW 5 major objections 4 minor 40 references

EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EvoHIL claims that keeping the reward, action, and visual interfaces adaptive together makes human-in-the-loop robot learning robust to lighting shifts.

desk verdict Transparent, well-built integration of reward adaptation, action-chunk flow policy, and relit offline replay, but the headline robustness claim rests on selected checkpoints and a tuned retention ratio; worth reviewing, not yet convincing. read the letter →

arxiv 2608.03872 v1 pith:P3BZ54GP submitted 2026-08-04 cs.RO

Shuoqin Zhang , Tongtong Cheng , Xiru Gao , Jinzhuo Peng , Bin Zheng , Jiahao Tu , Ke Wang , Jia Pan
show 2 more authors
Zhe Hu Kai Liu
This is my paper · ORCID
classification cs.RO
keywords human-in-the-loopreinforcementlearningrewardadaptationflowmatchingpolicyactionchunkingvisualdomainrelightingroboticmanipulationsuccessclassifier
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human-in-the-loop reinforcement learning lets robots learn contact-rich manipulation from limited real-world interaction, but the feedback loop usually leaves three interfaces frozen: the visual success classifier, the per-step action sampler, and the visual input distribution. EvoHIL is a staged framework that adapts all three within one actor-critic workflow: a self-evolving reward classifier is updated from human confirmations and provisional weak negatives, an action flow stabilizer replaces independent commands with flow-matched action chunks scored on their executed prefix, and a retention-aware offline phase replays relit recordings while anchoring the actor-critic to frozen references. The paper argues that these mechanisms are coupled only through the critic, preserving sparse terminal-success semantics. Across six manipulation tasks on two robot arms under a predefined illumination sweep, the complete pipeline reports the highest selected-policy success rates, with lower intervention rates, smoother normalized commands, and mostly shorter completion times than human-in-the-loop and imitation baselines.

What carries the argument

The framework rests on three named mechanisms. SER (self-evolving reward) is an online-updated binary success classifier whose training labels come only from human confirmations, with unconfirmed frames as weak negatives and a deployment threshold requiring K consecutive high-confidence frames before the classifier can emit a reward. AFS (action flow stabilization) is a flow-matching actor that generates action chunks and an execution-prefix critic that evaluates only the executed portion of each chunk; its FPO-style surrogate uses a conditional-flow-matching loss ratio, not an exact likelihood ratio, and is restricted to policy-generated segments. Retention-aware offline fine-tuning constru

What would settle it

Run retention-aware offline fine-tuning on relit videos where the relighter visibly deforms object edges or breaks frame-to-frame correspondence, then evaluate in a physically relit scene; if shifted-light success drops to the unanchored baseline level or intervention rates jump, the geometry-preservation assumption fails. A complementary check: evaluate the full pipeline on a lighting condition outside the predefined ten-step sweep, since the paper targets adaptation to the tested shift family rather than zero-shot generalization.

Watch

Extended reading notes

Core claim

The central claim is that the three deployment failures of human-in-the-loop reinforcement learning are not separate bugs but coupled limitations of a single fixed-interface loop, and that adapting reward, action, and visual domain within one actor-critic workflow solves them without changing the sparse reward semantics. Self-evolving reward (SER) updates the deployed success classifier from explicit human confirmations as trusted positives and unconfirmed frames as provisional weak negatives; label-source isolation prevents classifier predictions from becoming positive training labels, and a gated exponential-moving-average deployment limits reward drift. Action flow stabilization (AFS) lea

Load-bearing premise

The offline phase assumes that the external relighter changes only the appearance of the images and preserves the task geometry, object state, and temporal correspondence between consecutive frames; if relighting distorts any of these, the relit replay can train the actor-critic on transitions that never occurred in the real scene.

Editorial extensions

If this is right

  • Success classifiers deployed in human-in-the-loop systems can be kept aligned with the confirmation stream across scene changes without retraining from scratch or requiring new robot interaction.
  • Flow-matched action chunks with an execution-prefix critic yield smoother normalized commands than a single-step Gaussian actor, as shown by lower first/second differences and high-frequency power ratios.
  • Relit offline replay plus frozen-reference anchoring can transfer a policy to a shifted visual domain while retaining source-domain success, as the anchor-ablation and retention-ratio sweeps indicate.
  • Because SER and AFS interact only through the critic, each component can be enabled or ablated independently, and the same sparse terminal-success reward semantics are preserved.
  • If the relighter preserves geometry and frame correspondence, the visual-domain adaptation requires no additional human labels, expert actions, or robot rollouts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not stated in the paper: the label-source isolation rule generalizes beyond this setup; any human-feedback loop that treats only explicit positives as trusted and blocks model predictions from becoming positive labels can avoid self-reinforcing false positives.
  • We infer the retention-ratio and frozen-anchor scheme are transferable to other domain shifts such as texture, sensor noise, or background changes, provided the external relighter can be replaced by a suitable image-to-image transformation.
  • We infer that restricting the policy-gradient surrogate to executed prefixes may benefit other sparse-reward, contact-rich tasks, because the critic and the policy gradient evaluate exactly the commands the robot actually executed.
  • A testable extension: applying EvoHIL to a held-out illumination condition outside the predefined ten-combination sweep would separate robust adaptation from memorization of the evaluated shift family.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. EvoHIL proposes a staged human-in-the-loop RL framework that couples three adaptations: SER updates a terminal success classifier from human confirmations under label-source isolation; AFS replaces a Gaussian actor with flow-matched action chunks and an execution-prefix critic; retention-aware offline fine-tuning uses relit replay with frozen anchors to adapt to illumination shifts without new robot interaction. On six real-robot manipulation tasks across Franka FR3 and SO-101 under a predefined illumination sweep, the complete pipeline is reported to achieve the highest selected-policy success rates at the 60% operating point, with lower completion times and smoother commands than HIL-SERL, HG-DAgger, BC, IBRL, and ACT.

Significance. If the reported results hold, the integration is practically relevant: it addresses reward-model drift, action temporal coherence, and visual-domain shift within a single actor–critic workflow, and it does so with an unusually candid set of limitations. The label-source isolation in Eq. (8), the execution-prefix critic, and the retention anchors are useful design patterns, and the paper provides extensive real-hardware evaluation and component ablations. However, the central empirical claim currently rests on selected policies, a non-resource-matched baseline comparison, and an internal proxy metric, so the evidence is not yet strong enough for the abstract's unqualified 'improves' claim.

major comments (5)
  1. [Section V, Table IV; Section V-B-b] The headline success-rate comparison uses one selected policy per method, and the archived protocol does not document a separate checkpoint-selection set. In addition, α=0.75 was chosen as the best observed operating point in an α sweep on USB insertion (Section V-B-b), and USB insertion is one of the six scenes in Table IV. With each cell a 60-trial mean from a single training run and no training-run uncertainty, the claimed margins (e.g., 1.00 vs 0.50 for USB insertion) may reflect selection effects rather than robust improvement. Please add multiple independent runs with reported distributions, keep illumination/checkpoint selection on disjoint conditions, or present the conclusion as descriptive selected-policy evidence.
  2. [Section V-A and Section IV-B-b] Baselines are matched on demonstrations, real-robot interaction, and sensing, but only EvoHIL receives relit observations and the offline fine-tuning phase; the paper states these resources are not matched. This makes the comparison an end-to-end system comparison and prevents attributing the gains to any EvoHIL component. The central 'improves over baselines' claim needs a resource-matched control (e.g., a baseline given the same relit replay/offline updates, or an EvoHIL ablation without relit data) to show the advantage is not due to extra data/compute.
  3. [Section V-C-a and Eq. (8)] The main quantitative evidence for SER is balanced accuracy against the human-confirmation proxy, which is the same label stream used to train the classifier. Recovery after a scene change is therefore partly built into the training objective; the metric does not measure latent task-success accuracy. The paper acknowledges this, but the abstract lists 'agreement with human-confirmation labels' as a headline improvement. This should be reported as a proxy sanity check, with an external success label or task-level causal evidence needed to support reward-quality claims.
  4. [Section V-B, Tables I-II vs Section IV-B-c] The anchor ablations use the single-step Gaussian SAC backbone and frozen feature/Gaussian action-mean anchors, not the AFS execution-prefix critic anchor of Eq. (4) or the velocity-field anchor of Eq. (5) used in the final system. The paper states these results assess the replay/retention design rather than the AFS-specific anchor formulation. Consequently, the retention anchors in the actual EvoHIL pipeline are empirically unvalidated. Add ablations of Eqs. (4)-(5) in the full AFS system, or explicitly state that the transfer from the SAC anchors is assumed.
  5. [Section IV-B-a, Eq. (2)] Relit replay keeps actions, rewards, and terminations unchanged while replacing images. This is valid only under the geometry-preservation assumption: the relighter preserves task geometry, object state, and temporal correspondence between I_t and I_{t+1}. No fidelity check of the relighter is reported (Table VI records only cost and consistency). Since shifted-light success is the central result, a validity check is needed—e.g., comparing task-success labels before/after relighting or evaluating on relit-only policies—or the claim should be restricted to relighters with verified geometry preservation.
minor comments (4)
  1. [Section V-E, Fig. 12] BC and ACT are omitted from the full sweep; their 60% entries appear only in Table IV. A marker or annotation at the 60% point would help the reader integrate the two figures.
  2. [Abstract and Conclusion] The language alternates between 'improves' and 'descriptive selected-policy' caveats. The abstract should carry the same caveat as the conclusion, e.g., 'in these selected-policy evaluations'.
  3. [Section V-A] The statement that the complete system is 'independently evaluated' is qualified later. Move the qualification into the main text before Table IV so readers see it before the headline table.
  4. [Table III] Differences such as 0.93 vs 0.97 over 30 rollouts are within binomial resolution. State explicitly that these are not statistically distinguishable and are descriptive only.

Circularity Check

2 steps flagged · score 6.0 of 10

SER's recovery evidence is measured against the same human-confirmation labels used to train it, and the headline α/checkpoint choices are fitted on the evaluation scenes; the paper is transparent, but the central empirical claims are partly self-referential/fitted.

  1. self definitional [Section V-C-a (Fig. 7), Eq. (8), Section IV-C-b/c]
    "Every observed next frame is assigned the proxy target ˜yt =h + t . ... At each logging point, the metric is balanced accuracy computed on a random balanced sample of at most 256 frames from the accumulated live positive and provisional-negative buffers. The sample is neither temporally held out nor labeled independently for task success. ... It therefore measures agreement with the confirmation stream, not latent task-success accuracy."

    The recovery claim in Fig. 7 is computed on the same h^+ and provisional-negative label buffers that define the SER training targets in Eq. (8) and the loss in Eq. (10). The evaluation's positive and negative classes are literally the proxy classes used for training, and the sample may include training frames. So the self-evolving classifier's 'agreement' after a scene change is congruence with its own supervisory stream, not with an independent success signal. The held-out gate is also a split of the same proxy classes, so it does not add external validation. The paper's admission that the metric measures confirmation-stream agreement rather than latent task-success accuracy confirms the self-definitional character of this evidence.

  2. fitted input called prediction [Section V-B-b (Fig. 6), Eq. (3), Table IV]
    "The best observed operating point is therefore in the interior of the tested range, and the anchored objective has a wider high-performing region in this sweep. We useα= 0.75 for the reported EvoHIL experiments. ... Reported checkpoints were selected under the archived evaluation protocol, which does not document a separate set for checkpoint selection; the comparisons are therefore descriptive and may include checkpoint-selection effects."

    α is not derived from first principles; it is the best observed point in a sweep on USB insertion under original and shifted light (peaking at 1.00/1.00 success), and the same USB-insertion scene is then reported in Table IV at the 60% shift. Thus the Table IV margin (e.g., 1.00 vs 0.50 for HIL-SERL on USB insertion) is partly an artifact of selecting the hyperparameter on that scene. The qualification that reported checkpoints were selected without a separate selection set and that each cell is a 60-trial mean for one selected policy from one run means the 'strongest selected-policy success rates' in the abstract and contributions are optimized descriptive statistics, not held-out predictions. This is the fitted-input-called-prediction pattern.

full rationale

The paper does not derive its central result from a self-citation chain, an imported uniqueness theorem, or an ansatz smuggled in via citation; its methodological components are mostly assembled from external, independently developed building blocks. The main circularity is in the evidence used for two headline claims. First, the SER reward-classifier's recovery after scene changes is evaluated as balanced accuracy against the same human-confirmation/provisional-negative label stream that defines its training target in Eq. (8); the paper explicitly says this measures confirmation-stream agreement, not latent task-success accuracy, so the 'recovery' is at least partly self-measured. Second, the retention ratio α=0.75 is selected as the best operating point on USB insertion and then used in all EvoHIL experiments, including the USB-insertion row of Table IV, and checkpoints were selected without a separate held-out set; consequently the headline success rates are fitted/selected statistics rather than independent predictions. The paper is unusually transparent about both issues, and the internal label-source isolation in Eq. (8) does prevent a direct classifier-self-labeling loop; but the transparency does not remove the self-referential nature of the SER metric or the selection dependence of the empirical comparison. Score 6 reflects partial circularity in these central empirical claims, not a derivation that is entirely equivalent to its inputs.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a large set of manually chosen hyperparameters, most importantly the retention ratio alpha, which was selected by evaluating on the same task and condition used for the headline results. The method also relies on the external relighter preserving geometry and on human confirmations being trusted positives. The FPO surrogate is an unvalidated policy-improvement heuristic.

free parameters (9)
  • retention ratio alpha = 0.75
    Selected from a sweep on the USB insertion task under original and shifted lighting, using the same success metric as the main evaluation (Section V-B-b). This is a fitted operating point, not a principled value.
  • critic anchor weight lambda_Q = 0.2
    Set manually in Eq. 4; no ablation demonstrating a robust range is provided.
  • actor anchor weight lambda_pi = 0.1
    Set manually in Eq. 5; no ablation demonstrating a robust range is provided.
  • SER baseline consistency weight lambda_base = 0.1
    Manual choice in Eq. 10 affecting drift of the classifier.
  • SER negative-to-positive ratio = 5
    Manual choice for sampling provisional negatives; directly affects classifier bias.
  • SER balanced-accuracy acceptance threshold = 0.92
    Manual gate for deploying candidate classifiers.
  • SER EMA rate beta = 0.1
    Manual smoothing of deployed reward parameters.
  • AFS smoothness weight lambda_sm = 0.05
    Manual weight in Eq. 17 balancing temporal regularization.
  • task chunk/execution horizons H/E = 8/2, 16/4, 24/4 per task
    Chosen per task (Table V); affects temporal coherence and responsiveness of the flow actor.
assumptions (5)
  • domain assumption Relighter preserves task geometry, object state, and temporal correspondence between successive frames (Section IV-B-a).
    The relit replay in Eq. 2 is used as valid off-policy transitions under the shifted visual domain; if relighting changes geometry or correspondence, the offline adaptation trains on invalid data.
  • domain assumption Human confirmations are valid positives, i.e., h+ = 1 never occurs for failures (Section IV-C-b, Eq. 9).
    The SER update treats confirmations as trusted positive labels; the paper notes the confirmation-selection probability q(o) may be state-dependent, meaning the proxy is biased, but still assumes confirmations are true positives.
  • ad hoc to paper The FPO-style surrogate with clipped ratio exp(clip(l_ref - l_new, -2, 2)) is a usable policy-improvement objective (Section IV-D-d, Eqs. 14-15).
    The paper admits this is not an exact likelihood ratio, has no KL constraint or monotonic improvement guarantee, yet uses it as the core policy gradient for AFS.
  • standard math Standard off-policy actor-critic assumptions hold with equal replay/prior sampling (Section III-A).
    The framework inherits the standard MDP and off-policy learning assumptions for all updates.
  • standard math Conditional flow matching learns a valid velocity field for the action chunks (Section IV-D-a, Eq. 11).
    This is a direct application of the flow-matching framework from the cited literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning." pith.science (2026). https://pith.science/paper/P3BZ54GP

@misc{pith2026260803872,
  author       = {Pith},
  title        = {Pith review of: EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3BZ54GP}},
  note         = {Machine review of arXiv:2608.03872}
}
read the original abstract

Human-in-the-loop reinforcement learning (HIL-RL) enables robots to learn contact-rich manipulation from limited real-world interaction, but deployment exposes three coupled limitations: static visual reward models fail under scene changes; independently sampled actions cause temporally inconsistent motion; and vision-based policies remain sensitive to appearance shifts. We present EvoHIL, a unified framework that adapts the reward model, action generator, and visual do main within a staged human-in-the-loop learning process. First, self-evolving reward (SER) adapts the success classifier from human-confirmed positives and provisional weak negatives. Second, Action Flow Stabilization (AFS) generates temporally coherent action chunks through flow matching, grounding policy updates in executed action prefixes and demonstrated behavior. Third, retention-aware offline fine-tuning replays relit interaction data while anchoring the AFS actor-critic to prior behavior, adapting the visual domain without additional robot interaction. Across six manipulation tasks on Franka FR3 and SO-101 arms under a controlled lighting shift, EvoHIL improves task success, agreement with human-confirmation labels, motion smoothness, and completion time relative to human-in-the-loop and imitation baselines.Project page: https://anonymous4366.github.io/EvoHIL/

Figures

Figures reproduced from arXiv: 2608.03872 by the authors.

Figure 1
Figure 1. Six real-robot manipulation tasks across two embodiments. A 7-DoF Franka FR3 performs RAM insertion, USB insertion, table wiping, and circuit [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Staged overview of EvoHIL. During online interaction, SER updates the deployed success classifier from human confirmations, and AFS learns [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed architecture of AFS. The flow actor predicts an [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Camera placements for the six scenes. The FR3 tasks use wrist [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Success rate during fine-tuning on USB insertion (FR3) for the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: Agreement with human-confirmation proxy labels for the deployed [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 6
Figure 6. Figure 6: Sweep of the original-domain retention ratio [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 9
Figure 9. Figure 9: Illustrative execution trajectories at the [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Success rate and completion time for HIL-SERL, EvoHIL without [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: The ten ordered illumination conditions used for evaluation, spanning combinations of brightness, color temperature, shadow, and reflection changes. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Composite illumination-shift sweep (0% to 100%): success rate, intervention rate, and mean recorded episode duration for EvoHIL and the baselines across six scenes. All EvoHIL results use the unified SER–AFS pipeline and its required retention-aware offline phase. Eac…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 38 canonical work pages

  1. [1]

    Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning,

    J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning,”Science Robotics, vol. 10, no. 105, p. eads5033, 2025. 13

    work page 2025

  2. [2]

    SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning,

    J. Luo, Z. Hu, C. Xu, Y . L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine, “SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning,” inProc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024

    work page 2024

  3. [3]

    Efficient Online Reinforcement Learning with Offline Data,

    P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient Online Reinforcement Learning with Offline Data,” inProc. Int. Conf. Mach. Learn. (ICML), ser. PMLR, vol. 202, 2023, pp. 1577–1594

    work page 2023

  4. [4]

    Reinforcement Learning in Robotics: A Survey,

    J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement Learning in Robotics: A Survey,”The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013

    work page 2013

  5. [5]

    QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation,

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhoucke, and S. Levine, “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation,” inProc. Conf. Robot Learn. (CoRL), 2018, pp. 651–673

    work page 2018

  6. [6]

    How to Train Your Robot with Deep Reinforcement Learning: Lessons We Have Learned,

    J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to Train Your Robot with Deep Reinforcement Learning: Lessons We Have Learned,”The International Journal of Robotics Research, vol. 40, no. 4-5, pp. 698–721, 2021

    work page 2021

  7. [7]

    Addressing Function Approximation Error in Actor-Critic Methods,

    S. Fujimoto, H. van Hoof, and D. Meger, “Addressing Function Approximation Error in Actor-Critic Methods,” inProc. Int. Conf. Mach. Learn. (ICML), ser. PMLR, vol. 80, 2018, pp. 1587–1596

    work page 2018

  8. [8]

    Offline Reinforcement Learning with Implicit Q-Learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline Reinforcement Learning with Implicit Q-Learning,” inProc. Int. Conf. Learn. Represent. (ICLR), 2022

    work page 2022

Show all 40 references
  1. [9]

    A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,

    S. Ross, G. Gordon, and D. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” in Proc. Int. Conf. Artif. Intell. Stat. (AISTATS), ser. PMLR, vol. 15, 2011, pp. 627–635

    2011

  2. [10]

    HG-DAgger: Interactive Imitation Learning with Human Experts,

    M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive Imitation Learning with Human Experts,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2019, pp. 8077–8083

    2019

  3. [11]

    Deep Reinforcement Learning from Human Preferences,

    P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep Reinforcement Learning from Human Preferences,” inAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  4. [12]

    Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition,

    J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine, “Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition,” inAdvances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  5. [13]

    Positive-Unlabeled Reward Learning,

    D. Xu and M. Denil, “Positive-Unlabeled Reward Learning,” inProc. Conf. Robot Learn. (CoRL), ser. Proc. Mach. Learn. Res., vol. 155, 2021, pp. 205–219

    2021

  6. [14]

    Human-Guided Online Reward Adaptation for Real-Robot Arm Manipulation,

    T. Zhou, H. Ao, H. Lu, G. Chen, Z. Zhou, T. Cui, C. Yu, and Y . Yue, “Human-Guided Online Reward Adaptation for Real-Robot Arm Manipulation,”IEEE Robot. Autom. Lett., vol. 11, no. 8, pp. 9072–9079, 2026

    2026

  7. [15]

    On Calibration of Modern Neural Networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” inProc. Int. Conf. Mach. Learn. (ICML), 2017, pp. 1321–1330

    2017

  8. [16]

    Learning from Imbalanced Data,

    H. He and E. A. Garcia, “Learning from Imbalanced Data,”IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263–1284, 2009

    2009

  9. [17]

    A Survey on Concept Drift Adaptation,

    J. Gama, I. ˇZliobait˙e, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A Survey on Concept Drift Adaptation,”ACM Computing Surveys, vol. 46, no. 4, p. Article 44, 2014

    2014

  10. [18]

    Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,

    Y . Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V . Dil- lon, B. Lakshminarayanan, and J. Snoek, “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” in Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  11. [19]

    Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” inProc. Robot.: Sci. Syst. (RSS), 2023

    2023

  12. [20]

    Implicit Behavioral Cloning,

    P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit Behavioral Cloning,” inProc. Conf. Robot Learn. (CoRL), 2021, pp. 158–168

    2021

  13. [21]

    Flow Matching for Generative Modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow Matching for Generative Modeling,” inProc. Int. Conf. Learn. Represent. (ICLR), 2023

    2023

  14. [22]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,

    X. Liu, C. Gong, and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” inProc. Int. Conf. Learn. Represent. (ICLR), 2023

    2023

  15. [23]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” inProc. Robot.: Sci. Syst. (RSS), 2023

    2023

  16. [24]

    Reinforcement Learning with Action Chunking,

    Q. Li, Z. Zhou, and S. Levine, “Reinforcement Learning with Action Chunking,”arXiv preprint arXiv:2507.07969, 2025

    2025 arXiv

  17. [25]

    Diffusion Policy Policy Optimization,

    A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majum- dar, B. Burchfiel, H. Dai, and M. Simchowitz, “Diffusion Policy Policy Optimization,” inProc. Int. Conf. Learn. Represent. (ICLR), 2025

    2025

  18. [26]

    ReinFlow: Fine-Tuning Flow Matching Policy with Online Reinforcement Learning,

    T. Zhang, C. Yu, S. Su, and Y . Wang, “ReinFlow: Fine-Tuning Flow Matching Policy with Online Reinforcement Learning,” inAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  19. [27]

    Flow Matching Policy Gradients,

    D. McAllister, S. Ge, B. Yi, C. M. Kim, E. Weber, H. Choi, H. Feng, and A. Kanazawa, “Flow Matching Policy Gradients,” inProc. Int. Conf. Learn. Represent. (ICLR), 2026

    2026

  20. [28]

    SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Model- ing,

    Y . Zhang, S. Yu, T. Zhang, M. Guang, H. Hui, K. Long, Y . Wang, C. Yu, and W. Ding, “SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Model- ing,” inProc. Int. Conf. Learn. Represent. (ICLR), 2026

    2026

  21. [29]

    Reinforcement Learning with Augmented Data,

    M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement Learning with Augmented Data,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020

    2020

  22. [30]

    Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels,

    I. Kostrikov, D. Yarats, and R. Fergus, “Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021

    2021

  23. [31]

    Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World,

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World,” inProc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2017, pp. 23–30

    2017

  24. [32]

    AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty,

    D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty,” inProc. Int. Conf. Learn. Represent. (ICLR), 2020

    2020

  25. [33]

    Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control,

    NVIDIA, H. Abu Alhaija, J. Alvarez, M. Bala, T. Cai, T. Cao, L. Cha, J. Chen, M. Chen, F. Ferroni, S. Fidler, D. Fox, Y . Ge, J. Gu, A. Hassani, M. Isaev, P. Jannaty, S. Lan, T. Lasser, H. Ling, M.-Y . Liu, X. Liu, Y . Lu, A. Luo, Q. Ma, H. Mao, F. Ramos, X. Ren, T. Shen, X. S...

    2025

  26. [34]

    Experience Replay for Continual Learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience Replay for Continual Learning,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019

    2019

  27. [35]

    Regularizing Action Policies for Smooth Control with Reinforcement Learning,

    S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko, “Regularizing Action Policies for Smooth Control with Reinforcement Learning,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2021, pp. 1810–1816

    2021

  28. [36]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” inProc. Int. Conf. Mach. Learn. (ICML), 2018

    2018

  29. [37]

    Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping,

    A. Y . Ng, D. Harada, and S. Russell, “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping,” inProc. Int. Conf. Mach. Learn. (ICML), 1999

    1999

  30. [38]

    Imitation Bootstrapped Rein- forcement Learning,

    H. Hu, S. Mirchandani, and D. Sadigh, “Imitation Bootstrapped Rein- forcement Learning,” inProc. Robot.: Sci. Syst. (RSS), 2024

    2024

  31. [39]

    Deep Residual Learning for Image Recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778

    2016

  32. [40]

    UniRelight: Learning joint decomposition and synthesis for video relighting,

    K. He, R. Liang, J. Munkberg, J. Hasselgren, N. Vijaykumar, A. Keller, S. Fidler, I. Gilitschenski, Z. Gojcic, and Z. Wang, “UniRelight: Learning joint decomposition and synthesis for video relighting,”arXiv preprint arXiv:2506.15673, 2025

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.