Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Self-augmented feedback from the cheap GPT-4o-mini LLM can replace the privileged 'scripted teacher' in online preference-based RL, matching its success rates on MetaWorld tasks.

desk verdict The self-augmented feedback idea is genuinely new and worth a look, but the headline claim of matching a scripted teacher is not yet backed by numbers. read the letter →

arxiv 2412.16878 v1 pith:3DUPXATB submitted 2024-12-22 cs.LG cs.AI

Songjun Tu , Jingbo Sun , Qichao Zhang , Xiangyuan Lan , Dongbin Zhao This is my paper · ORCID
classification cs.LGcs.AI
keywords Onlinepreference-basedreinforcementlearningSelf-augmentedLLMfeedbackLLM-drivenQueryambiguityRewardfrompreferencesLargelanguagemodelsRoboticmanipulationMetaWorld
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to remove the two crutches of online preference-based reinforcement learning: the 'scripted teacher' that labels trajectory pairs by comparing privileged task rewards, and real-time human feedback that is impractical to obtain during training. It proposes RL-SaLLM-F, in which a large language model both judges which of two recent trajectories is closer to the task goal and imagines a better trajectory to use as extra preference data. The paper identifies 'query ambiguity' — early replay-buffer trajectories are so similar and suboptimal that LLM labels are barely above chance — and counters it with a double-check that swaps trajectory order and discards inconsistent labels, plus self-augmented imagined trajectories. On eight MetaWorld manipulation tasks, RL-SaLLM-F with GPT-4o-mini reaches success rates comparable to PEBBLE's scripted teacher and, on some tasks, matches SAC trained on the true reward. The upshot is a recipe for online PbRL that needs no privileged reward information and no human in the loop.

What carries the argument

The load-bearing mechanism is the preference dataset built from two kinds of LLM output. First, sampled trajectory pairs are converted into coordinate text and judged by the LLM; a double-check query with the two trajectories swapped retains a label only if both answers agree. Second, the LLM is prompted to generate an imagined trajectory that improves on the better of the two sampled trajectories while sharing its initial state, and the imagined trajectory is stored as preferred over the real one. These triples train a Bradley-Terry reward model, which relabels the replay buffer for an off-policy SAC agent. The imagined trajectories are admitted even when physically infeasible, on the assumption that the reward model is Markovian and therefore only needs the sequence of states for comparison, not for execution.

What would settle it

Take a trained RL-SaLLM-F reward model and compare its scores on a physically feasible expert trajectory and an LLM-generated trajectory that teleports the object to the target in one step; if the impossible trajectory receives a reward as high as or higher than the feasible one, the Markovian assumption is introducing bias. A direct experiment would train the same pipeline with deliberately infeasible but goal-reaching imagined trajectories and measure whether success rates collapse.

Watch

Extended reading notes

Core claim

The paper's central claim is that self-augmented LLM feedback can replace scripted-teacher feedback in online PbRL without sacrificing task performance. Concretely, RL-SaLLM-F is claimed to achieve comparable success rates to a scripted teacher with privileged reward information across eight MetaWorld tasks using only the lightweight GPT-4o-mini, while using no predefined reward and no human interaction. The paper also claims that the double-check mechanism lifts LLM label accuracy from about 58% to about 65%, and that adding LLM-generated imagined trajectories lifts it to about 72% while being the single largest contributor to policy success. It further claims that the imagined trajectories need not obey physical constraints, because they are used only to train a Markovian reward model through preference comparisons and never to drive the policy or a world model.

Load-bearing premise

The method stands on the assumption that the learned reward can be safely trained on LLM-imagined trajectories that may violate the environment's physics, so a reward model that is Markovian in state will not be misled by unrealistic states; the paper asserts this and supports it only with final success rates rather than a direct check.

Editorial extensions

If this is right

  • Online PbRL can be run with no privileged rewards: a single text-promptable LLM supplies both preference labels and extra training data.
  • Discarding ambiguous preference queries is better than labeling them as ties, since early ambiguous pairs are mostly hallucinations that harm reward learning.
  • The LLM's generative ability becomes a data-augmentation channel, improving label accuracy indirectly by making the policy sample more diverse, higher-quality trajectories.
  • The cost-performance tradeoff is controllable: GPT-4o-mini is roughly 20 times cheaper per query than GPT-4o and still matches the scripted teacher, while GPT-4o improves accuracy further.
  • Because the reward model only consumes state-coordinate text, the approach generalizes across tasks in the same state representation with only the goal description changed, and it transfers to a point-maze navigation benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the Markovian-reward justification implies that the self-augmentation channel would work with any generative trajectory source, not just an LLM; the paper's own Decision-Transformer comparison suggests the benefit depends on whether the generator produces goal-directed trajectories without expert data.
  • Editorial inference: query ambiguity is likely a property of early suboptimal replay data rather than of LLMs specifically, so the double-check-and-discard rule could serve as a general noise filter for any imperfect preference judge, including humans.
  • Editorial inference: using the scripted teacher as ground truth caps measured label accuracy; if the LLM's judgments were compared against human preferences instead, the 'label accuracy' numbers would be a different and arguably more meaningful quantity.
  • Editorial inference: the text-coordinate interface is the main practical constraint, so combining the framework with perception models that extract object coordinates from images, as the paper suggests as future work, is the natural path to vision-based tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RL-SaLLM-F, an online preference-based RL method that replaces the 'scripted teacher' with a large language model (GPT-4o-mini). The LLM provides preference labels for trajectory pairs from the replay buffer and additionally generates self-augmented 'imagined' trajectories that better achieve the task goal; a double-check mechanism discards trajectory pairs on which the LLM is order-inconsistent. Experiments on eight MetaWorld tasks claim comparable or better success rates than PEBBLE with scripted-teacher feedback and comparable to SAC with true task rewards, while using no privileged reward information. The paper also identifies a 'query ambiguity' failure mode in LLM-based preference discrimination during early online training.

Significance. If the empirical claims are validated, this work offers a practical direction for online PbRL that requires neither human feedback nor privileged reward information. The idea of using the same LLM both as a preference labeler and as a generator of augmented preference data is novel, and the choice of a lightweight, low-cost model (GPT-4o-mini) strengthens the applicability. The paper provides a public code link and includes several useful additional analyses: comparisons with reward-free VLM/DT baselines, an offline-dataset reward-model evaluation, point-maze experiments, and a cost analysis. However, the central 'comparable performance' claim currently rests only on learning curves without a final numerical table for the scripted-teacher baselines or statistical significance tests, and Remark 3 explicitly downgrades the comparison to 'reference only', which is in tension with the unqualified abstract claim.

major comments (4)
  1. [Section 6.2 / Abstract] The claim that RL-SaLLM-F achieves comparable performance to the scripted teacher is not quantitatively supported. Figure 3 shows only learning curves averaged over 5 seeds, and no final success-rate table with mean and standard deviation is provided for PEBBLE and SAC. Remark 3 explicitly calls the comparison 'reference only', yet the abstract states 'comparable performance' without qualification. Since this is the central contribution, provide a final performance table with per-task mean±std for all methods and a significance test (e.g., paired bootstrap or Wilcoxon signed-rank) to justify 'comparable', or soften the abstract accordingly.
  2. [Section 5.3, Remark 2] The assumption that physically infeasible imagined trajectories are harmless for reward learning is load-bearing but untested. The paper argues that a Markovian reward model makes infeasible states benign, but if the reward model assigns high rewards to states outside the feasible state manifold, the policy could be misled. Add a controlled experiment that constrains generated trajectories (e.g., by checking state-space bounds or using a simple forward model) and compare learning performance; also report whether generated trajectories stay within the range of states observed in real rollouts.
  3. [Section 6.2, Figure 3 / Table 4] The statistical evidence for the headline comparison is insufficient. With only 5 seeds and overlapping standard errors (e.g., Button Press and Drawer Open), the observed curve separations may be within noise. Table 4, which reports numerical results, does not include PEBBLE or SAC columns. Release per-seed results and final aggregate numbers for all baselines, and state clearly whether the observed differences are statistically reliable.
  4. [Section 6.5, Table 1 / Appendix B.8] Label accuracy is evaluated against the 'scripted teacher', which is itself a privileged-reward proxy, so the accuracy numbers do not directly measure agreement with human preferences. Additionally, the self-augmentation loop uses the same LLM to generate and label trajectories, which could create confirmation bias in the reward model. The offline-dataset experiment in Appendix B.8 is a good first step, but it measures only label accuracy, not policy performance; a human evaluation or a study of reward-model generalization to unseen trajectories would be more persuasive.
minor comments (6)
  1. [Section 6.6] The text says 'as shown in Figure 7' when referencing the bar-chart comparison of trajectory rewards; the correct reference is Figure 6.
  2. [Section 5.2] The double-check condition is specified only for the cases y=0, y_inv=1 and y=1, y_inv=0; it is unclear whether the case y=0.5, y_inv=0.5 (consistent 'equal preference' responses) is treated as valid or as a discard.
  3. [Section 6.5] The sentence 'Apart from the header, the first four rows' should read 'Apart from the header row'.
  4. [Appendix C.2] The label 'Onput 3' appears to be a typo for 'Output 3'.
  5. [Figure 1] The caption says 'Training curves of PEBBLE with LLM feedback' but the plot shows both LLM query accuracy and ground-truth episode rewards; consider clarifying the axes in the caption.
  6. [Section 5.1] The notation for the k-th nearest neighbor in the intrinsic reward is not defined; please clarify the distance metric and the value of k.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the central claim is empirical and is validated against external task rewards/success rates; the only self-citation is tangential and is not used to force any conclusion.

full rationale

The paper's central claim is not a first-principles derivation but an empirical result: RL-SaLLM-F attains comparable success rates to PEBBLE with scripted-teacher feedback on MetaWorld. The one loop that might look circular is the self-augmented feedback in Sec. 5.3: the LLM is prompted to generate a trajectory sigma_LLM that outperforms the better sampled trajectory, and the algorithm stores (sigma_LLM, sigma_{0/1}, y=0), labeling the generated trajectory as preferred. This makes each augmented training pair internally self-consistent by construction: the 'better' trajectory is, by definition, the one the LLM generated. However, this does not make the headline claim circular, because (i) the policy is executed in the real simulator and scored by task success rates, which are external to the LLM; (ii) Sec. 6.6 independently evaluates the LLM-generated trajectories using the environment's predefined reward and finds they score comparably to trajectories preferred by the scripted teacher; and (iii) Appendix B.8 validates the reward model on a public offline dataset, showing the learned preferences generalize beyond the LLM's self-generated labels. The only self-citation, [28] (Tu et al., 2024), appears once in Related Work as an example of a preference-noise challenge and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. Remark 2's reliance on physically infeasible imagined trajectories, Remark 3's 'reference only' disclaimer for the privileged-reward baselines, and the absence of final mean±std tables with only 5-seed curves are correctness/rigor concerns, not circularity: they affect whether the empirical comparison is fully supported, not whether any predicted quantity reduces by construction to its inputs. The paper is self-contained against external benchmarks, so the appropriate score is in the 0-2 range; 2 reflects the minor self-citation and the internal self-augmentation closed loop while the external evaluation keeps the central claim independent.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method introduces no new fitted parameters; all hyperparameters are standard (learning rate 0.0003, segment length 10, query budget 2000). The central claim rests on domain assumptions about the LLM's reliability and about Markovian reward learning from physically infeasible augmented trajectories.

assumptions (4)
  • domain assumption The reward model is Markovian, so step-wise rewards can be aggregated for preference comparisons, and physically infeasible imagined trajectories are harmless.
    Invoked in Remark 2, Section 5.3, to justify using LLM-generated trajectories that do not obey physical constraints.
  • domain assumption The LLM's preference judgments, after double-check, are accurate enough to train a useful reward model, and its generated trajectories are genuinely better as defined by the prompt.
    Central to Sections 5.2 and 5.3. The paper measures accuracy only against the scripted teacher proxy, which itself carries bias.
  • domain assumption The reduced state representation (TCP, object, and target coordinates) preserves sufficient information for reward learning and task success.
    Appendix B.1 describes the dimension reduction from 39 to 17 states. All methods use the same reduction, but the LLM's judgments are based entirely on this lossy text.
  • standard math The Bradley-Terry model correctly couples pairwise preferences to the reward model.
    Standard in PbRL, used in Equations (1) and (2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model." pith.science (2026). https://pith.science/paper/3DUPXATB

@misc{pith2026241216878,
  author       = {Pith},
  title        = {Pith review of: Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DUPXATB}},
  note         = {Machine review of arXiv:2412.16878}
}
read the original abstract

Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences. However, real-time human feedback is hard to obtain in online tasks. Most work suppose there is a "scripted teacher" that utilizes privileged predefined reward to provide preference feedback. In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL. RL-SaLLM-F leverages the reflective and discriminative capabilities of LLM to generate self-augmented trajectories and provide preference labels for reward learning. First, we identify an failure issue in LLM-based preference discrimination, specifically "query ambiguity", in online PbRL. Then LLM is employed to provide preference labels and generate self-augmented imagined trajectories that better achieve the task goal, thereby enhancing the quality and efficiency of feedback. Additionally, a double-check mechanism is introduced to mitigate randomness in the preference labels, improving the reliability of LLM feedback. The experiment across multiple tasks in the MetaWorld benchmark demonstrates the specific contributions of each proposed module in RL-SaLLM-F, and shows that self-augmented LLM feedback can effectively replace the impractical "scripted teacher" feedback. In summary, RL-SaLLM-F introduces a new direction of feedback acquisition in online PbRL that does not rely on any online privileged information, offering an efficient and lightweight solution with LLM-driven feedback.

Figures

Figures reproduced from arXiv: 2412.16878 by the authors.

Figure 1
Figure 1. Query ambiguities in online PbRL. (a) Two failed trajectories are provided, converted into text form, and input into [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of RL-SaLLM-F. First, trajectories are sampled from the replay buffer and converted into [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Learning curves of all compared methods on 8 tasks. Results are averaged over 5 seeds, and shaded regions represent [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Learning curves of the ablation study. When any component of RL-SaLLM-F is removed, the performance decreases. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Normalized step rewards for the expert and sub [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: Comparison of RL-SaLLM-F performance with GPT [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Illustration of partial tasks in MetaWorld. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Learning curves of all compared methods on 8 tasks, the vertical axis represents the trajectory return. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Learning curves of the ablation study, the vertical axis represents the trajectory returns. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Comparison of RL-VLM-F and RL-SaLLM-F. B.5 Comparison with Reward-free Baselines Since no direct baseline uses text descriptions, we compare with popular reward-free methods that learn rewards from observations using pretrained models. As suggested, we directly compar…
Figure 13
Figure 13. Figure 13: Learning curves of LLM and DT as trajectory generator. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A 3B VLM trained with SFT plus GRPO and an LCS-based reward reaches 55.3% on EmbodiedBench's EB-ALFRED, beating GPT-4o-mini and the 7B REBP planner.

Reference graph

Works this paper leans on

53 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Saurabh Arora and Prashant Doshi. 2021. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence 297 (2021), 103500

    work page 2021

  3. [3]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons.Biometrika 39, 3/4 (1952), 324–345

    1952

  4. [4]

    G Brockman. 2016. OpenAI Gym. arXiv preprint arXiv:1606.01540 (2016)

    arXiv 2016

  5. [5]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–15097

    2021

  6. [6]

    Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang. 2023. Robogpt: an intelligent agent of making embodied long-term decisions for daily instruction tasks. arXiv preprint arXiv:2311.15649 (2023)

    arXiv 2023

  7. [7]

    Yuanwen Chen, Xinyao Zhang, Yaran Chen, Dongbin Zhao, Yunzhen Zhao, Zhe Zhao, and Pengfei Hu. 2024. Common Sense Language-Guided Exploration and Hierarchical Dense Perception for Instruction Following Embodied Agents. In 2024 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 1–6

    work page 2024

  8. [8]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

Show all 53 references
  1. [9]

    Zibin Dong, Yifu Yuan, HAO Jianye, Fei Ni, Yao Mu, YAN ZHENG, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. 2024. AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model. In The Twelfth International Conference on Learning Representations

    2024

  2. [10]

    Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. 2024. FuRL: Visual- Language Models as Fuzzy Rewards for Reinforcement Learning. In Forty-first International Conference on Machine Learning

    2024

  3. [11]

    Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. 2022. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems 35 (2022), 15281–15295

    2022

  4. [12]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . PMLR, 1861– 1870

    2018

  5. [13]

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. 2024. Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning. InThe Twelfth International Conference on Learning Representations

    2024

  6. [14]

    Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. 2024. Query-Policy Misalignment in Preference-Based Reinforcement Learning. In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. 2021. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026 (2021)

    2021 arXiv

  8. [16]

    Kimin Lee, Laura M Smith, and Pieter Abbeel. 2021. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. In International Conference on Machine Learning. PMLR, 6152–6163

    2021

  9. [17]

    Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. 2022. Reward Un- certainty for Exploration in Preference-based Reinforcement Learning. In 10th International Conference on Learning Representations, ICLR 2022 . International Conference on Learning Representations

    2022

  10. [18]

    Calvin Luo, Mandy He, Zilai Zeng, and Chen Sun. 2024. Text-Aware Diffusion for Policy Learning. arXiv preprint arXiv:2407.01903 (2024)

    2024 arXiv

  11. [19]

    Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. 2023. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning . PMLR, 23301–23320

    2023

  12. [20]

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Eureka: Human-Level Reward Design via Coding Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  13. [21]

    Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. 2022. Zero-shot re- ward specification via grounded natural language. In International Conference on Machine Learning. PMLR, 14743–14752

    2022

  14. [22]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishin- skaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215 (2024)

    2024 arXiv

  15. [23]

    Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. 2022. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. In International Conference on Learning Representations

    2022

  16. [24]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology . 1–22

    2023

  17. [25]

    Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. 2024. Vision-Language Models are Zero-Shot Reward Models for Rein- forcement Learning. In The Twelfth International Conference on Learning Repre- sentations

    2024

  18. [26]

    Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. 2024. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems 36 (2024)

    2024

  19. [27]

    Richard S Sutton. 2018. Reinforcement learning: An introduction. A Bradford Book (2018)

    2018

  20. [28]

    Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. 2024. In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning.arXiv preprint arXiv:2412.09104 (2024)

    2024 arXiv

  21. [29]

    Belen Martin Urcelay, Andreas Krause, and Giorgia Ramponi. 2024. Reinforce- ment Learning from Human Text Feedback: Learning a Reward Model from Human Text Input. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment

    2024

  22. [30]

    David Venuto, Mohammad Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. 2024. Code as Reward: Empowering Reinforce- ment Learning with VLMs. In Forty-first International Conference on Machine Learning

    2024

  23. [31]

    Mudit Verma and Katherine Metcalf. 2024. Hindsight PRIORs for Reward Learning from Human Preferences. In The Twelfth International Conference on Learning Representations

    2024

  24. [32]

    Ruiqi Wang, Dezhong Zhao, Ziqin Yuan, Ike Obi, and Byung-Cheol Min. 2024. Prefclm: Enhancing preference-based reinforcement learning with crowdsourced large language models. arXiv preprint arXiv:2407.08213 (2024)

    2024 arXiv

  25. [33]

    Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. 2024. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. In Forty-first International Conference on Machine Learning

    2024

  26. [34]

    Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. 2024. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation. In Forty-first International Conference on ...

    2024

  27. [35]

    Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. 2017. A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research 18, 136 (2017), 1–46

    2017

  28. [36]

    Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. 2023. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309.11489

    2023 arXiv

  29. [37]

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. 2020. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning . PMLR, 1094–1100

    2020

  30. [38]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-Rewarding Language Models. In Forty-first International Conference on Machine Learning

    2024

  31. [39]

    Yuwei Zeng, Yao Mu, and Lin Shao. 2024. Learning Reward for Robot Skills Using Large Language Models via Self-Alignment. InForty-first International Conference on Machine Learning

    2024

  32. [40]

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al . 2024. Fine-Tuning Large Vision- Language Models as Decision-Making Agents via Reinforcement Learning. arXiv preprint arXiv:2405.10292 (2024)

    2024 arXiv

  33. [41]

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  34. [42]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  35. [43]

    Yupeng Zheng, Zhongpu Xia, Qichao Zhang, Teng Zhang, Ben Lu, Xiaochuang Huo, Chao Han, Yixian Li, Mengjie Yu, Bu Jin, et al . 2024. Preliminary In- vestigation into Data Scaling Laws for Imitation Learning-Based End-to-End Autonomous Driving. arXiv preprint arXiv:2412.02689 (2024)

    2024

  36. [44]

    Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, et al. 2024. PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning. arXiv preprint arXiv:2406.01587 (2024)

    2024 arXiv

  37. [45]

    scripted teacher

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. 2023. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In Interna- tional Conference on Machine Learning . PMLR, 43037–43067. Appendix A EXPERIMENTAL DETAILS A.1 Training details We traine...

    2023

  38. [47]

    tcp" represents the end position of the robot actuator, which is expressed in three-dimensional Cartesian coordinates in the range of [0,1]; (2)

    Which trajectory you think do better with achieving the goal? Reply a single line of 1 if you think the goal is better achieved in Trajectory 1, or 2 if it is better achieved in Trajectory 2. Reply 0 if the text is unsure or there is no significantly difference. Prompt for gen...

  39. [48]

    Is there any difference between Trajectory 1 and Trajectory 2 in terms of achieving the goal? Reply your analysis

  40. [49]

    Reply 0 if the text is unsure or there is no significantly difference

    Which trajectory you think do better with achieving the goal? Reply a single line of 1 if you think the goal is better achieved in Trajectory 1, or 2 if it is better achieved in Trajectory 2. Reply 0 if the text is unsure or there is no significantly difference. Output 1: Labe...

  41. [50]

    Object Position: The object is fixed at [-0.0229, 0.6803, 0.1150]

    Differences in Achieving the Goal: Trajectory 1: TCP Positions: The TCP moves from approximately [-0.0567, 0.8098, 0.4486] to [-0.1078, 0.7896, 0.4745]. Object Position: The object is fixed at [-0.0229, 0.6803, 0.1150]. Target Position: The target position is [-0.0229, 0.7739,...

  42. [51]

    tcp":[ [0.4363,0.8715,0.4302],[0.4358,0.8708,0.4314],[0.4355,0.8706,0.4304],[0.4355,0.8710,0.4280], ...... }; Trajectory 2: {

    Which Trajectory is Better? 2 Input 2: Label double-check # Same as input1, except the order of the tracks is swapped. ...... Trajectory 1: { "tcp":[ [0.4363,0.8715,0.4302],[0.4358,0.8708,0.4314],[0.4355,0.8706,0.4304],[0.4355,0.8710,0.4280], ...... }; Trajectory 2: { "tcp":[ ...

  43. [52]

    tcp":[ [0.4363,0.8715,0.4302], ],

    Which Trajectory is Better? 1 # Double-check: Pass Input 3: Better Trajectory Generation Based on your analysis, Can you generate a new trajectory based on the initial state of that good trajectory that you think can better achieve the goal? The generated trajectory should mee...

  44. [53]

    The 'obj' position remains constant at [0.0451,0.7300,0.0900] throughout the trajectory

    Analysis: Trajectory 1:* The 'tcp' positions gradually change over the 10 steps but remain relatively close to each other and are consistent in their path. The 'obj' position remains constant at [0.0451,0.7300,0.0900] throughout the trajectory. The 'target' is at [0.0451,0.540...

  45. [54]

    Therefore: *2*

    Result Based on the analysis, Trajectory 2 appears to make a more concerted effort to move towards the target 's Y-coordinate, albeit with some fluctuations in the path. Therefore: *2*

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.