Pith. sign in

REVIEW 3 major objections 6 minor 100 references

Rewriting the language prompt lets RL escape zero-success robot and coding policies that action noise cannot fix.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 06:20 UTC pith:YMVSPOFE

load-bearing objection Solid practical method for bootstrapping weak VLAs via prompt-space exploration; transfer to the canonical prompt is the real hinge, and the paper mostly shows it works. the 3 major comments →

arxiv 2607.08837 v1 pith:YMVSPOFE submitted 2026-07-09 cs.LG cs.AI

Prompt-Driven Exploration

Sunshine Jiang , John Marangola , David Zhang , Raghuram Kowdeed , Ruiyang Luo , Nitish Dashora , Richard Li , Pulkit Agrawal
show 1 more author
Zhang-Wei Hong
This is my paper
classification cs.LG cs.AI
keywords prompt-driven explorationreinforcement learningvision-language-action modelsposterior samplingprompt optimizationsample efficiencyrobot manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning stalls when a policy almost never succeeds: random action jitter only produces near-identical failures, so there is no reward signal to reinforce. This paper argues that language-conditioned policies (robot VLAs and LLMs) open a different exploration axis. Because the whole rollout follows from the natural-language prompt, changing the prompt can redirect the entire strategy. The method, Prompt-Driven Exploration (PDE), has a vision-language model watch the failed rollout videos, diagnose what the policy did, and rewrite the prompt to try something more useful next time. Those rewritten prompts act like posterior samples over policies; once they produce successful trajectories, ordinary RL transfers the skill back to the original task wording. On hard robot-manipulation suites and real Franka tasks where the starting policy scores near zero, PDE reaches high success under the original prompt while action-noise baselines stay near zero; the same idea also speeds early learning on coding and math LLM tasks.

Core claim

Prompt-Driven Exploration turns a vision-language model into an implicit posterior sampler over natural-language prompts. By diagnosing rollout videos and rewriting the prompt, PDE discovers globally different behaviors that action-space noise cannot reach, supplies the first successful trajectories from a zero-reward start, and lets RL fine-tuning improve the policy under the original evaluation prompt.

What carries the argument

Prompt-Driven Exploration (PDE): a VLM maintains an implicit distribution over prompts, updates it from compressed rollout summaries and success rates, samples exploratory prompts, and transfers the resulting successes back to the canonical prompt via mixture sampling and mixed backpropagation inside PPO (or analogous LLM RL algorithms).

Load-bearing premise

The method assumes a frozen vision-language model can reliably diagnose short rollout videos and propose prompts that unlock nonzero reward from a fixed weak policy, and that those successes will transfer back to the original evaluation wording.

What would settle it

On a held-out suite of hard LIBERO-PRO or real Franka tasks where the initial policy has zero success, run PDE versus action-noise PPO under identical rollout budgets: if the VLM-rewritten prompts never produce nonzero reward, or if success under the original prompt never rises while exploratory prompts succeed, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Prompt-Driven Exploration (PDE): for language-conditioned VLAs/LLMs, a VLM refines a distribution over natural-language prompts from rollout videos and success feedback, inducing global behavior changes that action-space noise rarely produces. The authors cast this as posterior sampling over an induced prompt-conditioned policy class (Alg. 1, §4.1), then transfer exploratory successes to the canonical evaluation prompt via mixture sampling and mixed backpropagation (Eq. 3, §4.2). Empirically, PDE is evaluated on a microwave case study, difficulty-stratified LIBERO-PRO (Pi0.5/GR00T/Pi0), ManiSkill OOD axes, three real Franka tasks, LiveCodeBench, and AIME, with ablations on random paraphrases, single-best prompts, and mixed backprop.

Significance. Exploration under near-zero success is a genuine bottleneck for VLA post-training; using the existing language interface as a structured exploration axis is a clean and timely idea. Strengths include breadth of evidence (simulation, real robot, LLM coding/math), interpretable prompt categories in the microwave study, difficulty-stratified aggregation, and ablations showing that unoptimized paraphrases and single-best-prompt training are insufficient (Fig. 9, Tables 8–9). If the transfer and budget claims hold under fair accounting, PDE is a practical contribution that other VLA RL pipelines can adopt without new density models or reward densifiers.

major comments (3)
  1. [§4.2 / App. A.2 / Figs. 4–6] Sample-budget accounting for the two-stage schedule (App. A.2) is load-bearing for the sample-efficiency claim. Prompt discovery freezes θ and spends T0×K×N rollouts per task before PPO; the microwave study alone uses ~85 discovery rollouts. Figures 4–6 and the abstract claim higher success with far fewer environment interactions, but it is unclear whether discovery rollouts are included on the Env Steps axis or only the RL stage. Please report total environment interactions (discovery + RL) for PDE and baselines on the same axes, or mark discovery cost separately, so the efficiency comparison is not understated for action-noise PPO.
  2. [§4.2 Eq. (3) / Table 2 / Tables 8–9] Transfer to the canonical prompt pg is the hinge of the central claim (successful policies under evaluation prompts from zero-reward starts), not discovery alone. Table 2 Regime 3 and Tables 8–9 already show discovery can be 0% while later RL under mixture+mixed backprop succeeds, and that training only on the best exploratory prompt collapses under pg (39.6% vs 63.8%). On hard zero-init tasks, please isolate transfer more cleanly: e.g., discovery-only (no θ update), discovery+PPO without mixed backprop (Eq. 3), and full PDE, all evaluated only under pg, with discovery cost counted. Without this, gains on hard tiers could be partly attributed to extra successful data under non-eval prompts rather than improved πθ(·|o,pg).
  3. [§4.1 / Abstract] The PSRL framing (§4.1) is an analogy: ρ is an implicit VLM sampler with no explicit density, likelihood, or Bayesian update, and history is compressed to one-sentence summaries plus success rates (App. A.1). That is fine as motivation, but phrases such as “realizes posterior sampling” and “amortized posterior update” overstate formality relative to classical PSRL. Please reframe as “PSRL-inspired prompt-space exploration” unless a concrete correspondence (e.g., what constitutes the likelihood and prior) is stated, so the contribution is not judged as a theoretical PSRL result.
minor comments (6)
  1. [Appendix tables] Table numbering in the appendix is inconsistent with in-text references (e.g., GR00T/Pi0 aggregate appears as Table 11 after Table 14 in the source order). Renumber tables in reading order.
  2. [§5.1 / App. A.5] Hyperparameters for the VLM supervisor (which model, temperature, frames_per_video) should be stated once in the main experimental setup, not only implied by templates in App. A.5.
  3. [Fig. 1 / §5.2] Figure 1 caption refers to improving from 10% to 50%, while the microwave narrative in §5.2 discusses ~0% to ~98%; align figure choice and caption with the section that cites it.
  4. [§5.3] Clarify whether language-perturbation tasks were excluded solely for confounding (§5.3) and whether that exclusion biases the hard-tier set toward tasks where prompt rephrasing is especially effective.
  5. [Eq. (3)] In Eq. (3), the denominator uses π_old under p while the numerator averages log-probs under pg and p; a one-sentence justification that this remains a valid importance ratio (or is a heuristic) would help readers.
  6. [§5.6 / Table 5] Real-world results (Fig. 8, Table 5) use only three tasks and 20 eval rollouts; report confidence intervals or binomial CIs so the “more than twice” claim is statistically readable.

Circularity Check

0 steps flagged

No circularity: PDE is an empirical exploration method evaluated on external environment rewards under held-out canonical prompts; the PSRL framing is an analogy, not a derivation that reduces to its inputs.

full rationale

The paper's load-bearing claims are empirical: that VLM-refined prompts unlock nonzero reward from weak VLAs/LLMs, and that mixture sampling plus mixed backpropagation (Eq. 3) transfers those successes so πθ(·|o, pg) improves under the canonical evaluation prompt. Success is defined by external task reward R(τp, g) that depends only on task completion, not on the prompt wording (Section 3), and is measured on LIBERO-PRO, ManiSkill, real Franka tasks, LiveCodeBench, and AIME against action-noise and other baselines under the same PPO/RLOO/GRPO budgets. The 'posterior sampling' language in §4.1 is explicitly an analogy: the authors state that ρ has 'no explicit density' and is 'queried as an implicit sampler,' not a derived Bayesian update. No parameter is fitted to data and then re-reported as a prediction; no uniqueness theorem or self-citation chain forces the method; ablations (random paraphrases, single-best prompt, mixed vs. original-only backprop) further test rather than define the result. The derivation chain therefore does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

PDE is an algorithmic construction, not a derivation from first principles. Load-bearing content is (1) standard RL/PSRL background, (2) the empirical assumption that pretrained VLAs/LLMs are strongly language-conditioned so prompt changes induce global coherent rollouts, (3) the empirical assumption that a VLM can diagnose failures from short video summaries, and (4) a handful of schedule and threshold hyperparameters that control the mixture and pool. No new physical entities are postulated.

free parameters (3)
  • mixture schedule α_t / consolidation target c = c=0.5, α_min=0.05, β=0.3
    Controls how quickly training shifts from exploratory prompts to the canonical prompt; default c=0.5, α_min=0.05; ablated but still hand-chosen.
  • pool admission threshold η and search budget (T0, K, N) = η>0, T0=10, K=5, N=10
    Determines which prompts enter the curriculum; η>0, T0=10, K=5, N=10 rollouts per candidate.
  • mixed-backpropagation log-prob weights = 1/2 + 1/2
    Equal 1/2 average of log π under exploratory and canonical prompts is a design choice, not derived.
axioms (4)
  • domain assumption For fixed VLA parameters, each natural-language prompt induces a coherent policy whose entire rollout can be globally shifted by rephrasing the prompt.
    Stated in §1 and §4.1; required for prompt-space exploration to be more than local action noise.
  • domain assumption A VLM can produce useful one-sentence diagnoses and rewrites from short rollout video summaries without gradient training on the target policy.
    Core of the implicit posterior update in §4.1 and Appendix A.1; if false, ρ does not concentrate on successful prompts.
  • domain assumption Reward depends only on task completion, not on prompt wording, so any prompt that elicits success is valid training data for the canonical-prompt policy.
    Explicit in §3; enables using exploratory prompts as RL data.
  • ad hoc to paper Classical PSRL exploration benefits transfer when the policy class is reparameterized as a prompt-conditioned family.
    Framing in §4.1; the paper does not prove regret bounds for the VLM-implicit posterior.
invented entities (1)
  • Prompt-Driven Exploration (PDE) / VLM-as-implicit-prompt-posterior independent evidence
    purpose: Names the algorithm that samples and updates a distribution over prompts from rollout feedback for RL exploration.
    The method is the paper's contribution; independent evidence is the empirical suite, not an external physical prediction.

pith-pipeline@v1.1.0-grok45 · 32621 in / 3283 out tokens · 29323 ms · 2026-07-13T06:20:46.010336+00:00 · methodology

0 comments
read the original abstract

Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original. Escaping a weak policy often requires global perturbations that action noise cannot produce. Large language models (LLMs) and vision-language-action (VLA) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes. The challenge is finding prompts that induce useful global changes. With a weak policy that rarely succeeds, reward is too sparse to select on. Our idea is to refine prompts from the rollouts themselves: a vision-language model (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time. This procedure realizes posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts, and improves sample efficiency more broadly. Our website is available at https://xinyunsunshine.github.io/prompt-rl.

Figures

Figures reproduced from arXiv: 2607.08837 by David Zhang, John Marangola, Nitish Dashora, Pulkit Agrawal, Raghuram Kowdeed, Richard Li, Ruiyang Luo, Sunshine Jiang, Zhang-Wei Hong.

Figure 1
Figure 1. Figure 1: Left: Given the prompt “put the green container on the bottom rack,” the VLA policy picks up the container but fails to place it fully on the rack, causing it to fall off the edge (top row). Rephrasing the prompt as “put the green container completely on the bottom rack” leads the same policy to succeed, without any weight updates (bottom row). Right: RL with our prompt-driven exploration (PDE) improves th… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our method. A VLM defines a prompt sampler [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: The SFT checkpoint fails under the original prompt “close the microwave”—the robot grasps the mug instead of pushing the door (top row). Our prompt optimization discovers an alternative prompt, “push on the microwave door until it shuts”, that redirects the policy to the correct contact point and action, achieving nonzero success without any weight updates (bottom row). Right: RL with our prompt driv… view at source ↗
Figure 4
Figure 4. Figure 4: Aggregated training curves grouped by task difficulty based on initial success rate. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Aggregated GR00T training curves grouped by task difficulty based on initial success rate. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Aggregated Pi0 training curves grouped by task difficulty based on initial success rate. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Real-world experi￾ment setup. Our hardware setup consists of a Franka FR3 arm, a set of ma￾nipulable tabletop objects, two drawers, and a two-level rack, as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Real-world manipulation results. (a) Average success rate over three real-world tasks. PPO+PDE improves substantially faster than PPO with action noise, achieving more than twice the success rate after 128 rollouts. (b) Failure modes of the original prompts and corresponding PDE-optimized prompts. The original prompts under-specify important task details, such as the target object or the required spatial m… view at source ↗
Figure 9
Figure 9. Figure 9: Ablation on prompt optimization (Object suite, task perturbation). PDE (optimized prompt pool) vs. Random Prompts (five un-optimized LLM paraphrases per task) vs. PPO (canonical prompt only). Curves are mean success rate across the 10 tasks; shaded regions are ±1 SE across tasks. Random paraphrases improve only marginally over PPO and fall well short of PDE, showing that optimizing which prompts enter the … view at source ↗
Figure 10
Figure 10. Figure 10: Cumulative unique nonzero prompts discovered vs. cumulative rollouts across three independent runs on the microwave task [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: AIME 2026 accuracy for GRPO and GRPO+PDE. Evaluation uses only the canonical [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: LiveCodeBench held-out accuracy for RLOO and PDE+RLOO. Training uses alternative [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

100 extracted references · 27 linked inside Pith

  1. [1]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Optimizing instructions and demonstrations for multi-stage language model programs , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  2. [3]

    Kevin Black and Noah Brown and James Darpinian and Karan Dhabalia and Danny Driess and Adnan Esmail and Michael Robert Equi and Chelsea Finn and Niccolo Fusai and Manuel Y. Galliker and Dibya Ghosh and Lachy Groom and Karol Hausman and brian ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Devin LeBlanc and Sergey Levine and Adrian Li-Bell an...

    2025

  3. [5]

    Foundations and Trends

    A tutorial on thompson sampling , author=. Foundations and Trends. 2018 , publisher=

    2018

  4. [6]

    Nature , volume=

    First return, then explore , author=. Nature , volume=. 2021 , publisher=

    2021

  5. [7]

    Advances in neural information processing systems , volume=

    Deep exploration via bootstrapped DQN , author=. Advances in neural information processing systems , volume=

  6. [8]

    Machine learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=

    1992

  7. [9]

    arXiv preprint arXiv:2511.15605 , year=

    SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models , author=. arXiv preprint arXiv:2511.15605 , year=

    arXiv

  8. [10]

    arXiv preprint arXiv:2505.18719 , year=

    Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning , author=. arXiv preprint arXiv:2505.18719 , year=

    Pith/arXiv arXiv

  9. [11]

    International Conference on Machine Learning , pages=

    Guiding pretraining in reinforcement learning with large language models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  10. [12]

    International conference on machine learning , pages=

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  11. [13]

    International conference on machine learning , pages=

    Curiosity-driven exploration by self-supervised prediction , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  12. [14]

    Advances in neural information processing systems , volume=

    Unifying count-based exploration and intrinsic motivation , author=. Advances in neural information processing systems , volume=

  13. [15]

    arXiv preprint arXiv:2310.12931 , year=

    Eureka: Human-level reward design via coding large language models , author=. arXiv preprint arXiv:2310.12931 , year=

    Pith/arXiv arXiv

  14. [19]

    Advances in Neural Information Processing Systems , volume=

    Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning , author=. Advances in Neural Information Processing Systems , volume=

  15. [24]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Fine-tuning and prompt optimization: Two great steps that work better together , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  16. [25]

    The eleventh international conference on learning representations , year=

    Large language models are human-level prompt engineers , author=. The eleventh international conference on learning representations , year=

  17. [26]

    The Twelfth International Conference on Learning Representations , year=

    Large language models as optimizers , author=. The Twelfth International Conference on Learning Representations , year=

  18. [28]

    International Conference on Machine Learning , pages=

    Liv: Language-image representations and rewards for robotic control , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  19. [29]

    The Thirteenth International Conference on Learning Representations , year=

    Vision language models are in-context value learners , author=. The Thirteenth International Conference on Learning Representations , year=

  20. [33]

    Fortunato and M

    M. Fortunato and M. G. Azar and B. Piot and J. Menick and I. Osband and A. Graves and V. Mnih and R. Munos and D. Hassabis and O. Pietquin and C. Blundell and S. Legg , title =. arXiv preprint arXiv:1706.10295 , year =

    Pith/arXiv arXiv

  21. [34]

    Plappert and R

    M. Plappert and R. Houthooft and P. Dhariwal and S. Sidor and R. Y. Chen and X. Chen and T. Asfour and P. Abbeel and M. Andrychowicz , title =. International Conference on Learning Representations (ICLR) , year =

  22. [35]

    Strens , title =

    M. Strens , title =. International Conference on Machine Learning (ICML) , year =

  23. [36]

    Osband and D

    I. Osband and D. Russo and B. Van Roy , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  24. [38]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=. doi:10.1038/s41586-025-09422-z , url=

    doi:10.1038/s41586-025-09422-z 2025

  25. [40]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  26. [41]

    The Twelfth International Conference on Learning Representations , year=

    Training Diffusion Models with Reinforcement Learning , author=. The Twelfth International Conference on Learning Representations , year=

  27. [42]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=

  28. [44]

    What Can

    Jijia Liu and Feng Gao and Bingwen Wei and Xinlei Chen and Qingmin Liao and Yi Wu and Chao Yu and Yu Wang , booktitle=. What Can. 2025 , url=

    2025

  29. [45]

    Haozhan Li and Yuxin Zuo and Jiale Yu and Yuhao Zhang and Yang Zhaohui and Kaiyan Zhang and Xuekai Zhu and Yuchen Zhang and Tianxing Chen and Ganqu Cui and Dehui Wang and Dingxiang Luo and Yuchen Fan and Youbang Sun and Jia Zeng and Jiangmiao Pang and Shanghang Zhang and Yu Wang and Yao Mu and Bowen Zhou and Ning Ding , booktitle=. Simple. 2026 , url=

    2026

  30. [46]

    2025 , note=

    RL Token: Bootstrapping Online RL with Vision-Language-Action Models , author=. 2025 , note=

    2025

  31. [47]

    Self-Improving Vision-Language-Action Models with Data Generation via Residual

    Wenli Xiao and Haotian Lin and Andy Peng and Haoru Xue and Tairan He and Zhengyi Luo and Yuqi Xie and Fengyuan Hu and Linxi Fan and Guanya Shi and Yuke Zhu , booktitle=. Self-Improving Vision-Language-Action Models with Data Generation via Residual. 2026 , url=

    2026

  32. [48]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Rlprompt: Optimizing discrete text prompts with reinforcement learning , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  33. [49]

    The Twelfth International Conference on Learning Representations , year=

    Large Language Models as Optimizers , author=. The Twelfth International Conference on Learning Representations , year=

  34. [50]

    Advances in Neural Information Processing Systems , volume=

    Ipo: Interpretable prompt optimization for vision-language models , author=. Advances in Neural Information Processing Systems , volume=

  35. [51]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Language models as black-box optimizers for vision-language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  36. [52]

    International journal of computer vision , volume=

    Learning to prompt for vision-language models , author=. International journal of computer vision , volume=. 2022 , publisher=

    2022

  37. [53]

    2026 , eprint=

    Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment , author=. 2026 , eprint=

    2026

  38. [55]

    Journal of artificial intelligence research , volume=

    Reinforcement learning: A survey , author=. Journal of artificial intelligence research , volume=

  39. [62]

    Machine learning , volume=

    Multitask learning , author=. Machine learning , volume=. 1997 , publisher=

    1997

  40. [63]

    Advances in neural information processing systems , volume=

    Distral: Robust multitask reinforcement learning , author=. Advances in neural information processing systems , volume=

  41. [64]

    , author=

    A practical guide to multi-objective reinforcement learning and planning: CF Hayes et al. , author=. Autonomous Agents and Multi-Agent Systems , volume=. 2022 , publisher=

    2022

  42. [65]

    International conference on machine learning , pages=

    Dynamic weights in multi-objective deep reinforcement learning , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  43. [66]

    2026 , eprint=

    Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs , author=. 2026 , eprint=

    2026

  44. [70]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  45. [71]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    Pith/arXiv arXiv 2023

  46. [72]

    Ahmadian, C

    A. Ahmadian, C. Cremer, M. Gall \'e , M. Fadaee, J. Kreutzer, O. Pietquin, A. \"U st \"u n, and S. Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12248--12267, 2024

    2024

  47. [73]

    Aky \"u rek, D

    E. Aky \"u rek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022

    Pith/arXiv arXiv 2022

  48. [74]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016

    2016

  49. [75]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \_0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 , 2024

    Pith/arXiv arXiv 2024

  50. [76]

    Black, M

    K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations , 2024

    2024

  51. [77]

    Brown et al

    T. Brown et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33, 2020

    2020

  52. [78]

    Burda, H

    Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018

    Pith/arXiv arXiv 2018

  53. [79]

    K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, Q. Zhang, Z. Yu, G. Fan, et al. rl: Online rl fine-tuning for flow-based vision-language-action models. arXiv preprint arXiv:2510.25889 , 2025

    arXiv 2025

  54. [80]

    J. R. Daniel, V. R. Benjamin, K. Abbas, O. Ian, and W. Zheng. A tutorial on thompson sampling. Foundations and Trends in Machine Learning , 11(1):1--99, 2018

    2018

  55. [81]

    Dekoninck, N

    J. Dekoninck, N. Jovanović, T. Gehrunger, K. Rögnvalddson, I. Petrov, C. Sun, and M. Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. 2026

    2026

  56. [82]

    Y. Du, W. Sun, and C. G. Snoek. Ipo: Interpretable prompt optimization for vision-language models. Advances in Neural Information Processing Systems , 37:126725--126766, 2024

    2024

  57. [83]

    Ecoffet, J

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature , 590(7847):580--586, 2021

    2021

  58. [84]

    D. Guo, D. Yang, Z. Zhang, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature , 645(8081), 2025

    2025

  59. [85]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861--1870. Pmlr, 2018

    2018

  60. [86]

    Z.-W. Hong, I. Shenfeld, T.-H. Wang, Y.-S. Chuang, A. Pareja, J. Glass, A. Srivastava, and P. Agrawal. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:2402.19464 , 2024

    Pith/arXiv arXiv 2024

  61. [87]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    Pith/arXiv arXiv 2024

  62. [88]

    Jaech, A

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024

    Pith/arXiv arXiv 2024

  63. [89]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024

    Pith/arXiv arXiv 2024

  64. [90]

    L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of artificial intelligence research , 4:237--285, 1996

    1996

  65. [91]

    Karnik, Z.-W

    S. Karnik, Z.-W. Hong, N. Abhangi, Y.-C. Lin, T.-H. Wang, C. Dupuy, R. Gupta, and P. Agrawal. Embodied red teaming for auditing robotic foundation models. arXiv preprint arXiv:2411.18676 , 2024

    Pith/arXiv arXiv 2024

  66. [92]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 , 2024

    Pith/arXiv arXiv 2024

  67. [93]

    J. Kwok, X. Zhang, M. Xu, Y. Liu, A. Mirhoseini, C. Finn, and M. Pavone. Scaling verification can be more effective than scaling policy learning for vision-language-action alignment, 2026

    2026

  68. [94]

    H. Li, Y. Zuo, J. Yu, Y. Zhang, Y. Zhaohui, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, D. Wang, D. Luo, Y. Fan, Y. Sun, J. Zeng, J. Pang, S. Zhang, Y. Wang, Y. Mu, B. Zhou, and N. Ding. Simple VLA - RL : Scaling VLA training via reinforcement learning. In The Fourteenth International Conference on Learning Representations , 2026

    2026

  69. [95]

    Liang, Y

    A. Liang, Y. Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, et al. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons. arXiv preprint arXiv:2603.02115 , 2026

    Pith/arXiv arXiv 2026

  70. [96]

    B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310 , 2023

    Pith/arXiv arXiv 2023

  71. [97]

    J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang. What can RL bring to VLA generalization? an empirical study. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025

    2025

  72. [98]

    Y. J. Ma, J. Hejna, C. Fu, D. Shah, J. Liang, Z. Xu, S. Kirmani, P. Xu, D. Driess, T. Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  73. [99]

    Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning , pages 23301--23320. PMLR, 2023

    2023

  74. [100]

    Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030 , 2022

    Pith/arXiv arXiv 2022

  75. [101]

    Opsahl-Ong, M

    K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 9340--9366, 2024

    2024

  76. [102]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016

    2016

  77. [103]

    Osband, D

    I. Osband, D. Russo, and B. V. Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3003--3011, 2013

    2013

  78. [104]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778--2787. PMLR, 2017

    2017

  79. [105]

    Plappert, R

    M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations (ICLR) , 2018

    2018

  80. [106]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

    Pith/arXiv arXiv 2017

Showing first 80 references.