Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that a handful of LLM-generated rollouts can warm-start an RL agent to reach near-optimal behavior with far fewer environment interactions than pure RL.

desk verdict The empirical recipe (pretrain on a small LLM-collected buffer, then fine-tune online) looks real and well-ablated; the theory section, however, leans on a guarantee that does not cover what LORO actually runs. read the letter →

arxiv 2505.10861 v1 pith:HTKJAXRJ submitted 2025-05-16 cs.LG

Thang Duong , Minglai Yang , Chicheng Zhang This is my paper · ORCID
classification cs.LG
keywords reinforcementlearninglargelanguagemodelssampleefficiencywarm-startingoffline-to-onlineRLMarkovdecisionprocessesLLM-guidedexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large language model's common-sense decisions can serve as a warm-start for classical reinforcement learning, cutting the number of environment interactions needed to learn a good policy. Its algorithm, LORO, first has the LLM roll out a small number of episodes, pre-trains a conventional RL policy on that data, and then fine-tunes the policy with online RL. The central claim is that under an assumption that the LLM's trajectories cover the state-action pairs an optimal policy visits, LORO both converges to an optimal policy and does so sample-efficiently. On six Gym environments, it reports beating pure RL, pure LLM policies, and a naive offline-to-online mixture, with up to four times the cumulative reward of the pure RL baseline.

What carries the argument

The load-bearing components are Assumption 1, which says the LLM policy's state-action coverage is sufficient for an optimal policy, and the LORO pipeline, which uses the LLM to collect tau episodes, pre-trains a policy on that data with DDQN for discrete actions and SAC for continuous actions, and then fine-tunes online without reusing the LLM buffer. The pre-training step is what separates LORO from a naive mix of offline and online data; the ablation studies show that skipping pre-training or replacing LLM data with random or on-policy data removes most of the benefit. The theoretical guarantee is inherited from an offline-to-online RL analysis that requires coverage, and LORO's sample-efficiency argument is that a good starting policy reduces unnecessary exploration.

What would settle it

Run LORO on a task where the optimal policy's visited state-action pairs are known, such as FrozenLake, and compare the LLM's trajectory coverage against that set; if an LLM policy with low coverage still yields LORO's gains, coverage is not the mechanism, and if high coverage is required for gains, tasks without it should fail.

Watch

Extended reading notes

Core claim

The core discovery is that an off-policy pre-training phase on a small dataset collected by an LLM, followed by on-policy RL, can outperform both pure online RL and naive data mixing. LORO's recipe is deliberately simple: use the LLM for the first tau episodes, train a value-based or actor-critic policy on those trajectories, then discard the LLM buffer and continue with online updates. The paper argues that the LLM's trajectories, though suboptimal, concentrate on regions that optimal policies visit, so the pre-trained policy begins close enough to avoid wasteful exploration while online RL provides the asymptotic optimality that the LLM alone lacks. Empirically, this yields the claimed factor-of-four cumulative-reward improvement over the on-policy baseline and, in four of six environments, convergence to near-optimal behavior.

Load-bearing premise

The paper's guarantee and intuitive story both rest on Assumption 1, that the LLM's rollouts sufficiently cover the state-action pairs an optimal policy would visit, and this assumption is never measured in any of the six environments.

Editorial extensions

If this is right

  • LORO attains up to four times the cumulative reward of the pure on-policy RL baseline on the six tested Gym environments, with learning curves that reach near-optimal performance in CartPole, FrozenLake, CliffWalking, and Pendulum.
  • Pre-training on the LLM-collected buffer is essential; simply mixing those trajectories into the online replay buffer performs roughly like pure RL.
  • The value of warm-starting depends on data quality: pre-training with random-policy or on-policy data does not reproduce the gains, so the LLM's coverage is the active ingredient.
  • The result suggests sample-efficient RL for tasks with expensive or risky environment interaction, since the environment only needs to be touched for the online refinement phase.
  • Increasing LLM size, Supervised Fine-Tuning, or Long Chain-of-Thought does not clearly improve the final policy, so a modest instruction-tuned model with chain-of-thought is enough.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of Assumption 1, measuring how often LLM rollouts visit state-action pairs that an optimal policy visits, would predict when LORO helps; the paper never performs this measurement.
  • The inherited theoretical guarantee assumes the algorithm keeps mixing offline data into updates, whereas LORO discards the LLM buffer after pre-training, so whether the guarantee transfers is an open question.
  • The same warm-start idea could be tried on sparse-reward tasks, where the LLM's subgoals or demonstrations might matter more, but dense-reward environments are the paper's scope.
  • If coverage degrades on larger state-action spaces, LORO's advantage may shrink; a natural extension is to measure coverage during data collection and adaptively augment the LLM buffer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LORO, an algorithm that has an LLM policy collect a small off-policy dataset, pre-trains an RL policy on that data, and then fine-tunes the policy online. Under an assumption that the LLM policy's trajectories sufficiently cover an optimal policy (Assumption 1), the authors claim that LORO inherits cumulative-suboptimality and sample-complexity guarantees from Song et al. (2022). Empirically, the paper reports results on six OpenAI Gym environments (CartPole, Pendulum, FrozenLake, CliffWalking, Represented Pong, MountainCar), comparing LORO against pure on-policy RL, pure LLM policies, and a mixing baseline, and reports that LORO achieves higher cumulative rewards, sometimes by a factor of about four over the RL baseline. Ablations examine the role of pretraining, the source of the warm-start data, LLM model size, number of pretraining steps, and amount of LLM data.

Significance. The paper addresses a timely and practically relevant question: whether LLM-generated rollouts can reduce the online interaction budget required for RL. The empirical study has notable strengths: six classic environments, five random seeds with standard error bars, and a series of ablations that isolate pretraining versus mixing, data source, model size, pretraining steps, and data quantity. The finding that LLM-data pretraining helps more than mixing the same data, and that random or on-policy pretraining data does not provide the same benefit, is a useful and falsifiable empirical result. The authors also make their code publicly available, which supports reproducibility. However, the paper's theoretical framing is not supported as stated: the cited Song et al. guarantee does not apply to the implemented pre-train-then-discard protocol, and Assumption 1 is never directly verified. The abstract's convergence claim is also stronger than the evidence, since two of the six environments are acknowledged not to converge to the optimal policy.

major comments (3)
  1. [Algorithm 1 / Appendix A.1.2 / §5.2] The central theoretical assertion, that under Assumption 1 LORO inherits the cumulative-suboptimality and sample-complexity guarantees of Song et al. (2022), does not follow from the cited result. Song et al.'s Hybrid RL algorithm keeps the offline dataset and mixes it with online data throughout; the authors themselves identify this mixing protocol in §5.2 as 'equivalent to Song et al. [2022]'. LORO instead pre-trains on the LLM-collected buffer and then, per Appendix A.1.2, 'only used online data in the online learning process,' meaning the offline buffer is discarded before fine-tuning. The cited theorem therefore does not apply to the implemented protocol. Moreover, Algorithm 1 line 14 continues to grow D with online data and line 12 passes D to Alg, which contradicts Appendix A.1.2; the paper must clarify which protocol was actually run and, if the Appendix is accurate, provide a new analysis for the pre-train-then-discard protocol.
  2. [Assumption 1 / §3 / Conclusion] Assumption 1 is the premise for the entire theoretical claim: πLLM must generate trajectories that sufficiently cover an optimal policy. The paper never measures or tests this coverage in any of the six environments; the authors instead infer it from the fact that LORO performs well empirically, which is a post hoc justification. The Conclusion explicitly concedes that 'Assumption 1 may not hold for some RL tasks.' Without an independent check—for example, estimating the state-action coverage of πLLM against the optimal policy's support—the theoretical claim remains conditional in an unverified way. The authors should either provide such a measurement or explicitly state that the theory is illustrative and not validated.
  3. [Abstract / §5.1 / Figure 2] The abstract and §5.1 state that LORO 'can both converge to an optimal policy and have a high sample efficiency,' but the experimental evidence does not support unconditional convergence. In the authors' own Figure 2 caption, the last two environments (MountainCar and RepresentedPong) are described as 'not converged to the optimal solution,' yet the abstract's claim is unqualified. The convergence claim should be restricted to the environments where it is observed, or the wording should be changed to reflect that the empirical benefit is improved cumulative reward and faster learning, not guaranteed convergence.
minor comments (5)
  1. [§2] There is a typo: 'high-performing enough enough' should be 'high-performing enough'.
  2. [§5.4 and Appendix B] The terms 'Majority V oting' and 'Best-of-N' contain a space in 'V oting'; please correct to 'Majority Voting' and 'Best-of-N' consistently.
  3. [Figure 2 caption] The caption says LORO reduces the data required 'from two to ten times' the vanilla On-Policy baseline, but no formal definition of this factor is given; specify how it is computed from the learning curves.
  4. [Appendix C.1.4] In the FrozenLake CoT example, the model's long reasoning argues for moving right, but the final output is 'Action: 1', which is inconsistent with the action mapping in the system prompt (1=left, 3=right). Please verify the example and the action mapping.
  5. [§5.2] The statement that 'pre-training significantly boosts the performance of LORO compared to just mixing' is based on a single configuration (τ=10, 1000 pretraining steps). State whether the conclusion holds across the full set of configurations reported in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the theoretical claim is conditional on an unverified assumption and a borrowed theorem whose applicability is questionable, which is a correctness gap rather than circularity.

full rationale

The paper does not derive its headline result from fitted parameters or from self-defined success criteria, and it does not rename a known empirical pattern as a new organization. The only theoretical engine is the external theorem of Song et al. (2022), invoked as a parameter-free guarantee under an explicit premise. That premise, Assumption 1, is stated as an input rather than an output: the authors write 'we create Assumption 1, which enables the LLM's collected dataset to satisfy Song et al. [2022], thus allowing our algorithm to enjoy their Cumulative suboptimality regret and Sample complexity guarantees.' This is conditional reasoning, not circular reasoning. The empirical benchmarks are external OpenAI Gym environments, and the claimed improvement over pure RL is measured there rather than defined in terms of the algorithm's own outputs. The paper's genuine weaknesses are support gaps, not circularity: (i) Assumption 1 is never measured in any environment and is acknowledged as a limitation in the conclusion; (ii) the cited Song et al. guarantee is for an algorithm that mixes offline and online data, while LORO discards the offline buffer after pretraining (Appendix A.1.2: 'we collected data to pre-train a policy and then only used online data in the online learning process'), and the paper itself says 'mixing the data alone (which is equivalent to Song et al. [2022])' is insufficient. These concerns undermine the unconditional wording of the abstract, but they do not make the derivation equivalent to its inputs by construction. Since no load-bearing step reduces to a self-citation, a fitted parameter renamed as a prediction, or a definitional identity, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

LORO has no invented entities. The algorithmic recipe is simple. The unverified coverage assumption and the transfer of an external regret bound are the main ledger items; hyperparameters are fixed by hand but ablated and are not fitted to the headline numbers.

free parameters (3)
  • tau (number of LLM warm-start episodes) = 10
    Chosen for the main runs (Section 5); Appendix A.4 varies it and finds no strong effect, so it is not tuned to maximize the headline result.
  • number of pretraining steps = 1000
    Chosen for main runs; Appendix A.3 shows 3000 gives similar results, so not fitted. Still a hand-selected hyperparameter.
  • LLM decoding settings = temperature 0.9, top-p 0.6, max 2000 tokens
    Fixed in Appendix B; arbitrary but not fitted to outcomes.
assumptions (3)
  • ad hoc to paper Assumption 1: trajectories following pi_LLM sufficiently cover an optimal policy (Section 3).
    This is the central enablement for the borrowed Song et al. guarantee; it is never verified empirically and is defined qualitatively, so if it fails the paper's warm-start benefit is not guaranteed.
  • ad hoc to paper Song et al. 2022's hybrid-RL regret bound transfers to LORO.
    The paper does not prove LORO satisfies the algorithm and dataset-usage conditions of Song et al.; in fact LORO removes the offline data after pretraining, while HyRL mixes offline and online data throughout. The guarantee is asserted, not derived.
  • domain assumption The MDP is a finite-horizon MDP with bounded rewards as in the Section 3 preliminaries.
    Standard RL formalization used implicitly; experiments use Gym environments, so it is plausible but unstated precisely.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM." pith.science (2026). https://pith.science/paper/HTKJAXRJ

@misc{pith2026250510861,
  author       = {Pith},
  title        = {Pith review of: Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTKJAXRJ}},
  note         = {Machine review of arXiv:2505.10861}
}
abstract

We investigate the usage of Large Language Model (LLM) in collecting high-quality data to warm-start Reinforcement Learning (RL) algorithms for learning in some classical Markov Decision Process (MDP) environments. In this work, we focus on using LLM to generate an off-policy dataset that sufficiently covers state-actions visited by optimal policies, then later using an RL algorithm to explore the environment and improve the policy suggested by the LLM. Our algorithm, LORO, can both converge to an optimal policy and have a high sample efficiency thanks to the LLM's good starting policy. On multiple OpenAI Gym environments, such as CartPole and Pendulum, we empirically demonstrate that LORO outperforms baseline algorithms such as pure LLM-based policies, pure RL, and a naive combination of the two, achieving up to $4 \times$ the cumulative rewards of the pure RL baseline.

Figures

Figures reproduced from arXiv: 2505.10861 by the authors.

Figure 1
Figure 1. The LLM Off-policy pre-train, RL On-policy (LORO) algorithm. Image inspired by Levine [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Our algorithm, LORO, outperforms the LLM policies (Qwen 7B, Qwen 32B) and the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparing pre-training (then removing the collected data) versus mixing the LLM’s [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Comparing pre-training with LLM’s data versus random and on-policy data. The main [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparing the effect of different amounts of pre-training data for the CartPole environment. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparing the effect of different LLMs’ model sizes for the CliffWalking environment. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Comparing the effect of different LLMs’ model sizes for the FrozenLake environment. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Comparing the effect of different LLMs’ model sizes for the CartPole environment. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Comparing the effect of different LLMs’ model sizes for the Pendulum environment. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Comparing the effect of different LLMs’ model sizes for the MountainCar environment. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Comparing the effect of different LLMs’ model sizes for the Pong environment. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Comparing the effect of different pre-training steps for the CliffWalking environment. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Comparing the effect of different pre-training steps for the FrozenLake environment. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Comparing the effect of different pre-training steps for the CartPole environment. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Comparing the effect of different pre-training steps for the Pendulum environment. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Comparing the effect of different pre-training steps for the MountainCar environment. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Comparing the effect of different pre-training steps for the Pong environment. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Comparing the effect of different amounts of pre-training data for the CliffWalking [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Comparing the effect of different amounts of pre-training data for the FrozenLake [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Comparing the effect of different amounts of pre-training data for the CartPole environ [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Comparing the effect of different amounts of pre-training data for the Pendulum environ [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Comparing the effect of different amounts of pre-training data for the MountainCar [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]
Figure 23
Figure 23. Figure 23: Comparing the effect of different amounts of pre-training data for the Pong environment. [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]
Figure 24
Figure 24. Figure 24: Comparing the effect of Long Chain-of-Thought and Supervised-Fine-Tuning for the [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]
Figure 25
Figure 25. Figure 25: The effect of Supervised-Fine-Tuning for the CliffWalking environment. [PITH_FULL_IMAGE:figures/full_fig_p023_25.png]
Figure 26
Figure 26. Figure 26: The effect of Supervised Fine-Tuning for the Pendulum environment. [PITH_FULL_IMAGE:figures/full_fig_p024_26.png]
Figure 27
Figure 27. Figure 27: FrozenLake with ineffective environment history. [PITH_FULL_IMAGE:figures/full_fig_p025_27.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ProDVI: Programmatic Dynamics Priors for Value Network Initialization

    cs.LG 2026-08 conditional novelty 6.0 of 10

    LLM-generated dynamics programs, used only to pretrain a value network's state-action encoder, improve sample efficiency of model-free RL on continuous control tasks.

Reference graph

Works this paper leans on

60 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Can language models encode perceptual structure without grounding? a case study in color

    Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders S gaard. Can language models encode perceptual structure without grounding? a case study in color. arXiv preprint arXiv:2109.06129, 2021

    arXiv 2021

  2. [2]

    Reinforcement learning: Theory and algorithms

    Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32: 0 96, 2019

    2019

  3. [3]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  4. [4]

    Unsupervised state representation learning in atari

    Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre C \^o t \'e , and R Devon Hjelm. Unsupervised state representation learning in atari. arXiv preprint arXiv:1906.08226, 2019

    arXiv 1906

  5. [5]

    Griffiths

    Dilip Arumugam and Thomas L. Griffiths. Toward efficient exploration by large language model agents, 2025. URL https://arxiv.org/abs/2504.20997

    arXiv 2025

  6. [6]

    Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine

    Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, 2023. URL https://arxiv.org/abs/2302.02948

    arXiv 2023

  7. [7]

    Neuro-dynamic programming: An overview and recent results

    Dimitri P Bertsekas. Neuro-dynamic programming: An overview and recent results. In Operations Research Proceedings 2006: Selected Papers of the Annual International Conference of the German Operations Research Society (GOR), Jointly Organized with the Austrian Society of Operations Research ( \"O GOR) and the Swiss Society of Operations Research (SVOR) Ka...

    work page 2006

  8. [8]

    Grounding llms for robot task planning using closed-loop state feedback

    Vineet Bhat, Ali Umut Kaypak, Prashanth Krishnamurthy, Ramesh Karri, and Farshad Khorrami. Grounding llms for robot task planning using closed-loop state feedback. arXiv preprint arXiv:2402.08546, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    Grounding large language models in interactive environments with online reinforcement learning

    Thomas Carta, Cl \'e ment Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676--3713. PMLR, 2023

    2023

  3. [11]

    Efficient sequential decision making with large language models

    Dingyang Chen, Qi Zhang, and Yinglun Zhu. Efficient sequential decision making with large language models. arXiv preprint arXiv:2406.12125, 2024

    2024 arXiv

  4. [12]

    Lmpriors: Pre-trained language models as task-specific priors

    Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task-specific priors. arXiv preprint arXiv:2210.12530, 2022

    2022 arXiv

  5. [13]

    Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks

    Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.01534, 2024

    2024 arXiv

  6. [14]

    Guiding pretraining in reinforcement learning with large language models

    Yuqing Du, Olivia Watkins, Zihan Wang, C \'e dric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pages 8657--8677. PMLR, 2023

    2023

  7. [15]

    Tree-based batch mode reinforcement learning

    Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005

    2005

  8. [16]

    Soft actor-critic algorithms and applications, 2019

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2019. URL https://arxiv.org/abs/1812.05905

    2019 arXiv

  9. [17]

    Planning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming

    Yilun Hao, Yang Zhang, and Chuchu Fan. Planning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming. arXiv preprint arXiv:2410.12112, 2024

    2024 arXiv

  10. [18]

    Deep q-learning from demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  11. [19]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 0 20482--20494, 2023

    2023

  12. [20]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  13. [21]

    Visual language maps for robot navigation

    Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10608--10615. IEEE, 2023

    2023

  14. [22]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022

    2022 arXiv

  15. [23]

    A survey of robot intelligence with large language models

    Hyeongyo Jeong, Haechan Lee, Changwon Kim, and Sungtae Shin. A survey of robot intelligence with large language models. Applied Sciences, 14 0 (19), 2024. ISSN 2076-3417. doi:10.3390/app14198868. URL https://www.mdpi.com/2076-3417/14/19/8868

    2024 doi

  16. [24]

    Bench llm deciders with gym translators

    Jarvis K. Bench llm deciders with gym translators. GitHub, 2024. URL https://github.com/mail-ecnu/Text-Gym-Agents

    2024

  17. [25]

    Housekeep: Tidying virtual households using commonsense reasoning

    Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal. Housekeep: Tidying virtual households using commonsense reasoning. In European Conference on Computer Vision, pages 355--373. Springer, 2022

    2022

  18. [26]

    Reinforcement learning in robotics: A survey

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32 0 (11): 0 1238--1274, 2013

    2013

  19. [27]

    Can large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024

    Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024

    2024 arXiv

  20. [28]

    Batch reinforcement learning

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pages 45--73. Springer, 2012

    2012

  21. [29]

    Supervised pretraining can learn in-context reinforcement learning

    Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 43057--43083, 2023

    2023

  22. [30]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  23. [31]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493--9500. IEEE, 2023

    2023

  24. [32]

    Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining

    Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXiv:2310.08566, 2023

    2023 arXiv

  25. [33]

    Physgen: Rigid-body physics-grounded image-to-video generation

    Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation. In European Conference on Computer Vision, pages 360--378. Springer, 2024

    2024

  26. [34]

    A survey of reinforcement learning informed by natural language

    Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rockt \"a schel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926, 2019

    1906 arXiv

  27. [35]

    Eureka: Human-level reward design via coding large language models

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023

    arXiv

  28. [36]

    Overcoming exploration in reinforcement learning with demonstrations

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292--6299. IEEE, 2018

    2018

  29. [37]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  30. [38]

    Evolve: Evaluating and optimizing llms for exploration

    Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024

    2024 arXiv

  31. [39]

    Llamagym: Fine-tune llm agents with online reinforcement learning

    Rohan Pandey. Llamagym: Fine-tune llm agents with online reinforcement learning. GitHub, 2024. URL https://github.com/KhoomeiK/LlamaGym

    2024

  32. [40]

    Mapping language models to grounded conceptual spaces

    Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International conference on learning representations, 2022

    2022

  33. [41]

    Deep q-learning: Theoretical insights from an asymptotic analysis, 2021

    Arunselvan Ramaswamy and Eyke Hüllermeier. Deep q-learning: Theoretical insights from an asymptotic analysis, 2021. URL https://arxiv.org/abs/2008.10870

    2021 arXiv

  34. [42]

    Neural fitted q iteration--first experiences with a data efficient neural reinforcement learning method

    Martin Riedmiller. Neural fitted q iteration--first experiences with a data efficient neural reinforcement learning method. In Machine learning: ECML 2005: 16th European conference on machine learning, Porto, Portugal, October 3-7, 2005. proceedings 16, pages 317--328. Springer, 2005

    2005

  35. [43]

    Kickstarting deep reinforcement learning

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835, 2018

    2018 arXiv

  36. [44]

    d3rlpy: An offline deep reinforcement learning library

    Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23 0 (315): 0 1--20, 2022. URL http://jmlr.org/papers/v23/22-0017.html

    2022

  37. [45]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  38. [46]

    It is too many options: Pitfalls of multiple-choice questions in generative ai and medical education, 2025

    Shrutika Singh, Anton Alyakin, Daniel Alexander Alber, Jaden Stryker, Ai Phuong S Tong, Karl Sangwon, Nicolas Goff, Mathew de la Paz, Miguel Hernandez-Rovira, Ki Yun Park, Eric Claude Leuthardt, and Eric Karl Oermann. It is too many options: Pitfalls of multiple-choice questio...

    2025 arXiv

  39. [47]

    Hybrid rl: Using both offline and online data can make rl efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718, 2022

    2022 arXiv

  40. [48]

    True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning

    Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning. arXiv preprint arXiv:2401.14151, 2024

    2024 arXiv

  41. [49]

    Gymnasium: A standard interface for reinforcement learning environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goul \ a o, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  42. [50]

    Representation learning for online and offline rl in low-rank mdps

    Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652, 2021

    2021 arXiv

  43. [51]

    Deep reinforcement learning with double q-learning, 2015

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, 2015. URL https://arxiv.org/abs/1509.06461

    2015 arXiv

  44. [52]

    Instabilities of offline rl with pre-trained neural representation

    Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham Kakade. Instabilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning, pages 10948--10960. PMLR, 2021

    2021

  45. [53]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  46. [54]

    Policy finetuning: Bridging sample-efficient offline and online reinforcement learning

    Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34: 0 27395--27407, 2021

    2021

  47. [55]

    Text2reward: Automated dense reward function generation for reinforcement learning

    Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309.11489, 2023

    2023 arXiv

  48. [56]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  49. [57]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  50. [58]

    Policy finetuning in reinforcement learning via design of experiments using offline data

    Ruiqi Zhang and Andrea Zanette. Policy finetuning in reinforcement learning via design of experiments using offline data. Advances in Neural Information Processing Systems, 36: 0 59953--59995, 2023

    2023

  51. [59]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of...

    2022

  52. [60]

    Language agent tree search unifies reasoning acting and planning in language models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.