{"id":"31d9251c-cbab-4e3d-8ab2-9c8086656fcd","arxiv_id":"2505.10861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Warm-starting RL with a small LLM-generated buffer and then fine-tuning online (LORO) improves cumulative reward over pure RL in six Gym environments, though the theoretical guarantee is borrowed and the coverage assumption unverified.","lead":"This paper tests whether starting reinforcement learning with a small dataset generated by a large language model, then fine-tuning online, cuts the samples needed to learn a good policy. In six classic control environments, the warmed-up learner generally beats both pure RL and pure LLM policies, sometimes by a large margin.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Song et al. 2022's guarantee is invoked but does not cover LORO's actual pre-train-then-discard protocol; Assumption 1 is never measured, so the convergence claim is unsupported.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test identifies a more specific defect than the unmeasured Assumption 1: the theoretical guarantee is not shown to be applicable to LORO's protocol. This is load-bearing because the paper's central claim about guaranteed convergence and sample efficiency rests on Song et al. (2022), and the protocol mismatch is an internal inconsistency that a reviewer cannot resolve without clarification. The empirical results cover only six classic environments, include two where LORO does not converge to the optimal policy, and use cumulative reward comparisons that are partly confounded by negative rewards. The concern is addressable: either retain the offline data during fine-tuning to match Song et al. and re-evaluate the need for pre-training, or derive a regret bound for the pre-train-then-discard scheme under a measured coverage condition. The reader's weakest assumption (unverified coverage) is necessary but not sufficient; even with coverage, the cited guarantee fails to transfer because the algorithms differ. Hence no change to the CONDITIONAL verdict is needed, and the recommended revisions should include resolving the Algorithm 1 / Appendix A.1.2 contradiction and either proving or qualifying the theoretical claim. I partially agree with the reader: the coverage assumption is indeed unmeasured, but the more direct obstacle is that the cited theorem does not cover the implemented algorithm.","tokens_in":27094,"tokens_out":6127,"duration_ms":61912,"concrete_test":"Run a tabular experiment to test the transfer: choose a small MDP (e.g., a 6-state grid) with a known optimal policy, construct an offline dataset from a suboptimal but covering πLLM (compute the coverage coefficient explicitly), and run three algorithms: (a) Song et al.'s Hybrid RL with the offline set retained in the replay buffer, (b) LORO per Algorithm 1 (buffer grows with online data), and (c) LORO per Appendix A.1.2 (offline data discarded after pre-training). Compare cumulative suboptimality and sample complexity against the rate promised by Song et al. If (b) or (c) violate the bound while (a) respects it, the paper must either prove a new bound for the pre-train-then-discard scheme or soften the theoretical claim; if all three match, the paper should clarify the exact buffer management and cite the applicable theorem accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical assertion—that under Assumption 1 LORO inherits the cumulative suboptimality and sample-complexity guarantees of Song et al. (2022)—does not follow from the cited result. Song et al.'s Hybrid RL is a mixing algorithm: it retains the offline dataset and incorporates it into online updates, a point the authors themselves make in §5.2 where 'mixing the data alone' is called 'equivalent to Song et al. [2022]'. LORO, however, pre-trains on the LLM-collected buffer and then, per Appendix A.1.2, 'only used online data in the online learning process'—the offline buffer is discarded before fine-tuning. The cited theorem therefore does not apply to the implemented protocol, and a new analysis would be required. The paper is internally inconsistent: Algorithm 1 line 14 continues to grow D with online data, while Appendix A.1.2 says only online data is used, so it is unclear which algorithm was actually run. Additionally, even the premise for any such guarantee, Assumption 1 (πLLM's state-action coverage of an optimal policy), is never measured or tested in any environment; the authors acknowledge this limitation in the Conclusion. Consequently, the abstract's unconditional claim that LORO 'can both converge to an optimal policy and have a high sample efficiency' is not backed by the cited theory. The empirical trends on six Gym tasks are suggestive but do not substitute for the missing guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LORO, an algorithm that has an LLM policy collect a small off-policy dataset, pre-trains an RL policy on that data, and then fine-tunes the policy online. Under an assumption that the LLM policy's trajectories sufficiently cover an optimal policy (Assumption 1), the authors claim that LORO inherits cumulative-suboptimality and sample-complexity guarantees from Song et al. (2022). Empirically, the paper reports results on six OpenAI Gym environments (CartPole, Pendulum, FrozenLake, CliffWalking, Represented Pong, MountainCar), comparing LORO against pure on-policy RL, pure LLM policies, and a mixing baseline, and reports that LORO achieves higher cumulative rewards, sometimes by a factor of about four over the RL baseline. Ablations examine the role of pretraining, the source of the warm-start data, LLM model size, number of pretraining steps, and amount of LLM data.","tokens_in":27412,"tokens_out":4595,"duration_ms":42267,"significance":"The paper addresses a timely and practically relevant question: whether LLM-generated rollouts can reduce the online interaction budget required for RL. The empirical study has notable strengths: six classic environments, five random seeds with standard error bars, and a series of ablations that isolate pretraining versus mixing, data source, model size, pretraining steps, and data quantity. The finding that LLM-data pretraining helps more than mixing the same data, and that random or on-policy pretraining data does not provide the same benefit, is a useful and falsifiable empirical result. The authors also make their code publicly available, which supports reproducibility. However, the paper's theoretical framing is not supported as stated: the cited Song et al. guarantee does not apply to the implemented pre-train-then-discard protocol, and Assumption 1 is never directly verified. The abstract's convergence claim is also stronger than the evidence, since two of the six environments are acknowledged not to converge to the optimal policy.","major_comments":[{"comment":"The central theoretical assertion, that under Assumption 1 LORO inherits the cumulative-suboptimality and sample-complexity guarantees of Song et al. (2022), does not follow from the cited result. Song et al.'s Hybrid RL algorithm keeps the offline dataset and mixes it with online data throughout; the authors themselves identify this mixing protocol in §5.2 as 'equivalent to Song et al. [2022]'. LORO instead pre-trains on the LLM-collected buffer and then, per Appendix A.1.2, 'only used online data in the online learning process,' meaning the offline buffer is discarded before fine-tuning. The cited theorem therefore does not apply to the implemented protocol. Moreover, Algorithm 1 line 14 continues to grow D with online data and line 12 passes D to Alg, which contradicts Appendix A.1.2; the paper must clarify which protocol was actually run and, if the Appendix is accurate, provide a new analysis for the pre-train-then-discard protocol.","section":"Algorithm 1 / Appendix A.1.2 / §5.2"},{"comment":"Assumption 1 is the premise for the entire theoretical claim: πLLM must generate trajectories that sufficiently cover an optimal policy. The paper never measures or tests this coverage in any of the six environments; the authors instead infer it from the fact that LORO performs well empirically, which is a post hoc justification. The Conclusion explicitly concedes that 'Assumption 1 may not hold for some RL tasks.' Without an independent check—for example, estimating the state-action coverage of πLLM against the optimal policy's support—the theoretical claim remains conditional in an unverified way. The authors should either provide such a measurement or explicitly state that the theory is illustrative and not validated.","section":"Assumption 1 / §3 / Conclusion"},{"comment":"The abstract and §5.1 state that LORO 'can both converge to an optimal policy and have a high sample efficiency,' but the experimental evidence does not support unconditional convergence. In the authors' own Figure 2 caption, the last two environments (MountainCar and RepresentedPong) are described as 'not converged to the optimal solution,' yet the abstract's claim is unqualified. The convergence claim should be restricted to the environments where it is observed, or the wording should be changed to reflect that the empirical benefit is improved cumulative reward and faster learning, not guaranteed convergence.","section":"Abstract / §5.1 / Figure 2"}],"minor_comments":[{"comment":"There is a typo: 'high-performing enough enough' should be 'high-performing enough'.","section":"§2"},{"comment":"The terms 'Majority V oting' and 'Best-of-N' contain a space in 'V oting'; please correct to 'Majority Voting' and 'Best-of-N' consistently.","section":"§5.4 and Appendix B"},{"comment":"The caption says LORO reduces the data required 'from two to ten times' the vanilla On-Policy baseline, but no formal definition of this factor is given; specify how it is computed from the learning curves.","section":"Figure 2 caption"},{"comment":"In the FrozenLake CoT example, the model's long reasoning argues for moving right, but the final output is 'Action: 1', which is inconsistent with the action mapping in the system prompt (1=left, 3=right). Please verify the example and the action mapping.","section":"Appendix C.1.4"},{"comment":"The statement that 'pre-training significantly boosts the performance of LORO compared to just mixing' is based on a single configuration (τ=10, 1000 pretraining steps). State whether the conclusion holds across the full set of configurations reported in the appendix.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The reader's and skeptic's core concern is valid: the theoretical guarantee is invoked for a protocol that differs from the one in the cited theorem, and the paper's own appendix confirms the discrepancy. The empirical study is solid enough to be salvageable, but the abstract and theoretical sections need substantial rework, not just copyediting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical recipe in this paper—collect a small buffer with an LLM policy, pre-train an RL algorithm on it, then fine-tune online—looks like it works on these six Gym environments, and the ablations support the main claim. Treat the experimental part as the contribution. The advertised convergence guarantee, on the other hand, is not established.\n\nWhat the paper does well: it runs five seeds on six environments, compares against pure LLM policies, on-policy RL, and random policies, and includes ablations that isolate the source of the gain. The finding that mixing the LLM data into online learning is not enough, while pre-training on it first is, is a clean and useful empirical observation. So is the result that pre-training only helps with LLM-quality data, not random or on-policy data. They also release code, which makes the experiments reproducible.\n\nSoft spots, in rough order of severity. First, the theory does not transfer. Song et al. 2022 analyzes a mixing algorithm that keeps the offline dataset in the online phase. LORO pre-trains on the LLM buffer and then, per Appendix A.1.2, discards it and uses only online data. The paper even says that mixing alone is equivalent to Song et al. and then shows mixing does not work. So the cited theorem cannot be doing the work the abstract claims. Second, Assumption 1—that the LLM policy covers the state-action pairs of an optimal policy—is never measured or tested in any environment. The authors acknowledge this in the conclusion, but it is the entire basis for any theoretical anchor. Without that, the convergence and sample-complexity claims are unfounded. Third, there is an internal inconsistency: Algorithm 1 line 14 keeps appending online data to the buffer D, while Appendix A.1.2 says only online data is used. It is unclear which protocol was actually run, and the distinction matters for the theory. These are fixable problems, but they need to be fixed.\n\nMinor points: the environments are simple, and some LLM baselines are very weak (Qwen gets 0 on FrozenLake), so the empirical gains are suggestive rather than general. The \"up to 4x\" headline depends heavily on environment and baseline; several plots show LORO improving but not converging. Still, these are secondary.\n\nThe paper is for empirical RL practitioners who want to warm-start with LLM rollouts. I would not cite it yet for the theory, but I would point someone to it as evidence that pre-training on LLM data can beat mixing. It deserves a serious referee because the empirical recipe is plausible and potentially useful, and the flaws are addressable. My recommendation: send it to review, and ask the authors to either prove an appropriate guarantee for the pre-train-then-discard protocol or explicitly drop the theoretical claim, and to clean up the algorithm-buffer inconsistency.","headline":"The empirical recipe (pretrain on a small LLM-collected buffer, then fine-tune online) looks real and well-ablated; the theory section, however, leans on a guarantee that does not cover what LORO actually runs.","tokens_in":27944,"tokens_out":2210,"would_cite":false,"duration_ms":23675,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a handful of LLM-generated rollouts can warm-start an RL agent to reach near-optimal behavior with far fewer environment interactions than pure RL.","keywords":["reinforcement learning","large language models","sample efficiency","warm-starting","offline-to-online RL","Markov decision processes","LLM-guided exploration"],"falsifier":"Run LORO on a task where the optimal policy's visited state-action pairs are known, such as FrozenLake, and compare the LLM's trajectory coverage against that set; if an LLM policy with low coverage still yields LORO's gains, coverage is not the mechanism, and if high coverage is required for gains, tasks without it should fail.","tokens_in":26860,"feed_emoji":"🤖","tokens_out":6529,"duration_ms":54875,"temperature":0.7,"pith_summary":"The paper sets out to show that a large language model's common-sense decisions can serve as a warm-start for classical reinforcement learning, cutting the number of environment interactions needed to learn a good policy. Its algorithm, LORO, first has the LLM roll out a small number of episodes, pre-trains a conventional RL policy on that data, and then fine-tunes the policy with online RL. The central claim is that under an assumption that the LLM's trajectories cover the state-action pairs an optimal policy visits, LORO both converges to an optimal policy and does so sample-efficiently. On six Gym environments, it reports beating pure RL, pure LLM policies, and a naive offline-to-online mixture, with up to four times the cumulative reward of the pure RL baseline.","feed_headline":"Few LLM rollouts lift RL rewards up to 4x","feed_subtitle":"LORO pre-trains on a handful of LLM episodes, then online RL refines the policy with far fewer environment steps.","key_machinery":"The load-bearing components are Assumption 1, which says the LLM policy's state-action coverage is sufficient for an optimal policy, and the LORO pipeline, which uses the LLM to collect tau episodes, pre-trains a policy on that data with DDQN for discrete actions and SAC for continuous actions, and then fine-tunes online without reusing the LLM buffer. The pre-training step is what separates LORO from a naive mix of offline and online data; the ablation studies show that skipping pre-training or replacing LLM data with random or on-policy data removes most of the benefit. The theoretical guarantee is inherited from an offline-to-online RL analysis that requires coverage, and LORO's sample-efficiency argument is that a good starting policy reduces unnecessary exploration.","core_discovery":"The core discovery is that an off-policy pre-training phase on a small dataset collected by an LLM, followed by on-policy RL, can outperform both pure online RL and naive data mixing. LORO's recipe is deliberately simple: use the LLM for the first tau episodes, train a value-based or actor-critic policy on those trajectories, then discard the LLM buffer and continue with online updates. The paper argues that the LLM's trajectories, though suboptimal, concentrate on regions that optimal policies visit, so the pre-trained policy begins close enough to avoid wasteful exploration while online RL provides the asymptotic optimality that the LLM alone lacks. Empirically, this yields the claimed factor-of-four cumulative-reward improvement over the on-policy baseline and, in four of six environments, convergence to near-optimal behavior.","pith_inferences":["A direct test of Assumption 1, measuring how often LLM rollouts visit state-action pairs that an optimal policy visits, would predict when LORO helps; the paper never performs this measurement.","The inherited theoretical guarantee assumes the algorithm keeps mixing offline data into updates, whereas LORO discards the LLM buffer after pre-training, so whether the guarantee transfers is an open question.","The same warm-start idea could be tried on sparse-reward tasks, where the LLM's subgoals or demonstrations might matter more, but dense-reward environments are the paper's scope.","If coverage degrades on larger state-action spaces, LORO's advantage may shrink; a natural extension is to measure coverage during data collection and adaptively augment the LLM buffer."],"forward_implications":["LORO attains up to four times the cumulative reward of the pure on-policy RL baseline on the six tested Gym environments, with learning curves that reach near-optimal performance in CartPole, FrozenLake, CliffWalking, and Pendulum.","Pre-training on the LLM-collected buffer is essential; simply mixing those trajectories into the online replay buffer performs roughly like pure RL.","The value of warm-starting depends on data quality: pre-training with random-policy or on-policy data does not reproduce the gains, so the LLM's coverage is the active ingredient.","The result suggests sample-efficient RL for tasks with expensive or risky environment interaction, since the environment only needs to be touched for the online refinement phase.","Increasing LLM size, Supervised Fine-Tuning, or Long Chain-of-Thought does not clearly improve the final policy, so a modest instruction-tuned model with chain-of-thought is enough."],"supporting_citations":[{"why":"Supplies the offline-to-online cumulative-suboptimality and sample-complexity guarantee that LORO inherits under Assumption 1.","marker":"Song et al. [2022]"},{"why":"Motivates the problem by showing that naive offline pretraining plus online fine-tuning can hurt, which LORO's pretraining design addresses.","marker":"Nair et al. [2020]"},{"why":"Provides the LlamaGym codebase and LLM-action design that the experiments build on.","marker":"Pandey [2024]"},{"why":"The d3rlpy library supplies the RL algorithm implementations and default hyperparameters used in pretraining and online learning.","marker":"Seno and Imai [2022]"},{"why":"DDQN is the off-policy RL algorithm used for all discrete-action environments.","marker":"van Hasselt et al. [2015]"},{"why":"SAC is the continuous-action algorithm used for Pendulum.","marker":"Haarnoja et al. [2019]"},{"why":"Qwen2.5-7B and Qwen2.5-32B Instruct are the LLM policies that collect the warm-start data.","marker":"Yang et al. [2024]"},{"why":"Gymnasium provides the six benchmark environments used in evaluation.","marker":"Towers et al. [2024]"},{"why":"Frames the grounding problem that motivates combining LLM decisions with online RL.","marker":"Carta et al. [2023]"},{"why":"Chain-of-thought prompting is the mechanism by which the LLM produces its actions.","marker":"Wei et al. [2022]"}],"fun_headline_variants":["LLM warm-start gives RL 4x reward boost","Few LLM demos quadruple RL sample efficiency","LLM pre-training makes RL 4x more data-efficient","LLM kickstarts RL to 4x rewards with few demos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's guarantee and intuitive story both rest on Assumption 1, that the LLM's rollouts sufficiently cover the state-action pairs an optimal policy would visit, and this assumption is never measured in any of the six environments.","fun_headline_variants_meta":{"raw":{"variants":["LLM warm-start gives RL 4x reward boost","Few LLM demos quadruple RL sample efficiency","LLM pre-training makes RL 4x more data-efficient","LLM kickstarts RL to 4x rewards with few demos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2629,"prompt_tokens":868,"completion_tokens":1761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1690}},"tokens_in":484,"tokens_out":1761,"duration_ms":12343,"temperature":1.0,"reasoning_tokens":1690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:01:44.231242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LORO on a task where the optimal policy's visited state-action pairs are known, such as FrozenLake, and compare the LLM's trajectory coverage against that set; if an LLM policy with low coverage still yields LORO's gains, coverage is not the mechanism, and if high coverage is required for gains, tasks without it should fail.","supporting_citations":[{"cited_title":"Llamagym: Fine-tune llm agents with online reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the LlamaGym codebase and LLM-action design that the experiments build on."},{"cited_title":"Grounding large language models in interactive environments with online reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Frames the grounding problem that motivates combining LLM decisions with online RL."}],"review_version":1}