{"id":"7444fe4a-2f30-422c-934a-68c6277246ba","arxiv_id":"2505.00234","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Agents that accumulate their own successful trajectories as retrieval examples gain up to 20 points on ALFWorld, Wordcraft, and InterCode-SQL, with two curation methods pushing gains further.","lead":"This paper shows that LLM agents can improve their own success rates on sequential tasks by saving their successful attempts and reusing them as in-context examples, without extra prompt engineering. Across three benchmarks this self-bootstrapping lifts success rates by up to 20 points, rivaling the gains from a more powerful model or multiple attempts per task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never tests whether self-generation or the success filter is the active ingredient; gains could come from any large trajectory database.","rationale":"I read the paper as a claim that an agent can bootstrap its own performance by accumulating its successful trajectories as in-context examples. The core numbers are plausible and the paper includes useful ablations: the best/worst exemplar comparison, pass@k context, and fine-tuning results. However, the experimental design does not isolate the proposed mechanism. The baseline is a small fixed hand-written database; the treatment adds many self-generated successful trajectories, changing database scale, source, and filtering simultaneously. Without a control using an externally sourced database of similar size, or unfiltered trajectories, the specific contribution of self-generation is untested. The reader's focus on a reliable success signal is a special case of this gap: if the filter is unnecessary or if any successful trajectories work, the method's dependence on a verifier is not the key limitation. I therefore agree partially with the reader: the concern is real but better characterized as a missing attribution control, not only an environmental assumption. The recommended check would settle it; if the external-source control matches Traj-Bootstrap, the verdict should remain conditional but the central claim would need to be reframed.","tokens_in":19692,"tokens_out":14302,"duration_ms":163466,"concrete_test":"On ALFWorld, construct a static retrieval database by running Fixed-DB (or a scripted oracle) on the 3,500 training tasks, keeping successful trajectories only, without any online bootstrapping or curation. Evaluate the same ReAct agent with this externally sourced database on the same 134 test tasks and compare to Traj-Bootstrap's 0.89. If the externally sourced database matches or exceeds 0.89, the improvement is not specific to self-generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that self-generated successful trajectories drive the gains—is underdetermined by the experiments. The only comparison is Fixed-DB, a small hand-written database (18 examples on ALFWorld), versus Traj-Bootstrap, which grows the same retrieval database with up to 3,500 agent-generated successful trajectories (Sec. 6.1.3, Table 1). This conflates three variables: database size/diversity, the success filter, and the self-generated source. There is no control that populates the database with (a) all trajectories including failures, or (b) successful trajectories generated by an external, non-bootstrapping policy. If either condition performs as well as Traj-Bootstrap, then the self-bootstrapping-from-own-successes mechanism is not the cause; the result would simply be that more successful examples help in-context retrieval, which is already known from many-shot ICL work (e.g., Agarwal et al. 2024). The best/worst exemplar analysis (Fig. 4) shows that trajectory quality matters once candidates exist, but it does not establish that the candidates must be self-generated. Appendix E shows bootstrapping from an empty database improves on Wordcraft, which is evidence for self-generation, but it still does not compare against an external source of trajectories. Because the title and abstract make the self-generated property central, this missing control is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes trajectory bootstrapping as a way to build retrieval databases of in-context examples for ReAct-style LLM agents in sequential decision-making tasks. Starting from a small set of human-provided exemplars, Traj-Bootstrap appends only successful self-generated training trajectories to the database, which is queried at each decision point with a multi-key embedding retrieval scheme. On three benchmarks (ALFWorld, Wordcraft, InterCode-SQL), naive accumulation raises held-out success rates from 73% to 89%, from 55% to 64%, and from 75% to 79%, respectively. Two curation mechanisms are then introduced: population-based database-level curation (+DB-Curation) and exemplar-level curation (+Exemplar-Curation) based on a retrieval-weighted quality metric Q(τ). The combined method reaches 93% on ALFWorld and 82% on InterCode-SQL. The paper contextualizes the gains against test-time sampling (pass@k), LLM upgrades, fine-tuning on the collected data, and prior systems (AutoManual, Autoguide).","tokens_in":19935,"tokens_out":18174,"duration_ms":172454,"significance":"If the central claim is established, the paper contributes a practically appealing, task-agnostic mechanism for agent self-improvement that avoids prompt engineering, and the reported gains (e.g., matching pass@2–pass@3 under a single test-time attempt) are meaningful. The manuscript has genuine strengths: evaluations use five seeds with clean train/test separation (App. F.2); the pass@k comparison in Table 3 is a fair and interpretable baseline; Fig. 4's best-versus-worst exemplar comparison is a good causal check on the curation metric; and App. E directly tests whether the method can bootstrap from an empty database. The fine-tuning results in App. C indicate the collected data have independent value. However, the central causal claim that self-generation, rather than database size or the success filter, drives the gains is not isolated by the experiments; the paper's own invocation of many-shot in-context learning results (refs. [8, 22]) highlights this gap. The headline contextual claims in the abstract (exceeding model upgrades, surpassing AutoManual) are also stronger than the measurements support.","major_comments":[{"comment":"The central claim that self-generated trajectories are the active mechanism is underdetermined by the experiments. The only comparison offered is Fixed-DB (18 hand-written trajectories on ALFWorld, per §H.1) versus Traj-Bootstrap (up to 3,500 agent-generated successful trajectories), which jointly changes database size and diversity, the success filter of §5.1, and the source of the trajectories. No control populates the database with (a) all trajectories including failures, or (b) successful trajectories from an external, non-bootstrapping policy (for example, a stronger LLM or an independent agent). Without (b), the gains are consistent with the known many-shot in-context-learning effect that the authors themselves invoke in §6.2 (refs. [8, 22]); without (a), the contribution of the success filter is never isolated. Appendix E shows that bootstrapping from an empty database still scales with database size on Wordcraft, which is partial evidence for self-generation, but it still does not compare against an external source of successful trajectories. Because the title and abstract make the self-generated property the mechanism, this missing source control is load-bearing; I recommend adding both controls on at least ALFWorld and Wordcraft and reporting whether the Traj-Bootstrap gains survive.","section":"§5.1, §6.2, Table 1"},{"comment":"The abstract claims that the reported improvements 'exceed those achieved by upgrading from gpt-4o-mini to gpt-4o,' but this comparison is shown only for ALFWorld (Table 2), and even there the difference is not statistically meaningful: Traj-Bootstrap improves 0.73 to 0.89 (16 points) while the model upgrade improves 0.73 to 0.88 (15 points), with overlapping standard errors. The §6.3 sentence stating that the 20-point boost 'significantly outperforms' the 15-point upgrade is not supported by any significance test, and the 20-point figure itself appears to be an arithmetic error (0.73 to 0.89 is 16 points; 0.73 to 0.93 for the combined method is 20 points). The abstract and §6.3 should be revised to state the ALFWorld-only comparison with appropriate uncertainty.","section":"§6.3, Table 2, Abstract"},{"comment":"The 'surpassing' comparison with AutoManual in §6.3 and the abstract uses very different training budgets: +DB+Exemplar-Curation is trained on 3,500 tasks (with five parallel instances under +DB-Curation), while the AutoManual rows in Table 2 use 36 tasks. As reported, the comparison bundles algorithmic merit with a roughly two-order-of-magnitude difference in environment interactions, so 'surpassing approaches that use more powerful LLMs and hand-crafted components' overstates what is established. Please report the number of LLM calls (the paper gives totals only in App. I, not per method) and either match the baseline's training budget or evaluate both methods at 36, 100, and 3,500 tasks so the scaling behavior is visible.","section":"§6.3, Table 2, Abstract"}],"minor_comments":[{"comment":"The abstract's 'without human intervention' overstates what the method requires: §3 assumes an initial set of human-generated trajectories and a descriptor of the action space, and Appendix E shows that removing the human examples permanently lowers Wordcraft performance even after 4,000 collected trajectories; please qualify this phrase.","section":"Abstract, §3"},{"comment":"The method relies on a reliable binary success signal at training time, which is stated in §2.1 and used as the storage filter in §5.1; the paper should explicitly discuss robustness to noisy or unavailable success signals, since contamination of the database by failed trajectories would likely remove the reported benefit.","section":"§2.1, §5.1"},{"comment":"Equation (2) in Appendix B.4 restates Equation (1) from §5.3; please cross-reference instead of redefining the metric, and decide on a single numbering.","section":"§5.3, B.4"},{"comment":"The benchmark-specific choices (category-based similarity on ALFWorld; appending the solution query to the goal on InterCode-SQL) soften the claim of avoiding task-specific engineering; clarify whether these follow the prior conventions of [10] and [37], and test whether the InterCode-SQL result persists without the goal-append modification.","section":"§H.1, §H.2"},{"comment":"Please state the sampling temperature used for the Fixed-DB pass@k attempts in Table 3; the k-attempt equivalence is only meaningful if the sampling distribution matches the temperature (0.1) used for the single-attempt evaluations.","section":"§6.3, Table 3"},{"comment":"There are several typographical errors, including 'Traj-Boostrap' in the Table 2 caption and 'appraoches' in Appendix A, and the benchmark name appears both as 'Intercode-SQL' and 'InterCode-SQL'; please normalize these.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reports a clearly executed empirical study with an honest evaluation protocol, and I would be comfortable seeing it published after the missing source control is added and the abstract is aligned with the actual measurements. The main risk is novelty: if the gains are attributable to database size rather than self-generation, the contribution reduces to an agentic instance of many-shot in-context learning with dynamic retrieval. I would suggest ensuring one of the added control experiments (external-source trajectories) is included in the revision, and asking the authors to release code and trajectory databases to allow verification (the API cost is approximately $3,000 with about 2.7M calls, per App. I)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: this is a solid, mostly well-executed empirical paper showing that accumulating successful trajectories into a retrieval database substantially improves LLM-agent success on three benchmarks. The headline result, that naive accumulation alone lifts ALFWorld from 73% to 89%, is real and worth knowing. But the paper's central claim—that self-generation is the active ingredient—is not actually tested, and the abstract overstates some comparisons.\n\nWhat's new: prior work like ExpeL and AutoGuide already learn from self-generated experience, but they rely on abstraction or rule extraction. The authors show that simple accumulation of successful trajectories works, and they study two curation mechanisms: population-based database selection and a retrieval-weighted quality metric. Both are sensible and give modest gains. The pass@k comparison is clean and fair, and the paper is honest about variance and diminishing returns. The appendix ablation bootstrapping from an empty database on Wordcraft is a nice addition.\n\nSoft spots: the biggest one is the missing control. Traj-Bootstrap is compared only against Fixed-DB, a small hand-written database. This conflates three variables: database size, the success filter, and the self-generated source. There is no condition that stores all trajectories, including failures, or stores successful trajectories from an external policy. Without that, the results do not distinguish 'self-generated successful examples help' from 'more successful examples help,' which is already known from many-shot ICL. The best/worst exemplar analysis (Fig. 4) shows that trajectory quality matters once you have candidates, but not that the candidates must be self-generated. The empty-DB ablation is evidence for self-generation, but still lacks an external-trajectory control.\n\nSecond, the abstract overclaims. The 'exceeds model upgrade' result is ALFWorld-only, and the AutoManual comparison uses 3500 training tasks against AutoManual's 36. The paper is clearer in Table 2, but the abstract does not carry those caveats.\n\nThird, reproducibility: no code and the exact initial human trajectories are not included. On ALFWorld they use ExpeL's set, but for the other two they just say 'collected.' Five seeds is fine, but the exact prompts and initial examples are needed to verify.\n\nMinor: the reliable success signal assumption is standard in these sparse-reward environments, so I would not count it as a flaw.\n\nWho it's for: anyone working on LLM agents or in-context learning will get value from the empirical results. The curation methods could be useful, though their gains are modest. The paper deserves a serious referee; the missing control is a significant gap but not a deal-breaker. I'd ask the authors to add the external-success and all-trajectory controls, or, at minimum, soften the self-generation claim.\n\nRecommendation: send it to review, with the missing control as the main revision point.\n\nBest,\n[You]","headline":"A clean empirical study of bootstrapping retrieval databases, but the missing control leaves the 'self-generated' claim underdetermined.","tokens_in":20487,"tokens_out":3597,"would_cite":true,"duration_ms":35188,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that LLM agents can improve on new sequential-decision tasks by accumulating their own successful trajectories and using them as in-context examples.","keywords":["LLM agents","in-context learning","trajectory bootstrapping","sequential decision-making","retrieval-augmented agents","self-improvement","data curation","population-based training"],"falsifier":"Corrupt the success signal during training, for example by flipping the stored success bit for 20% of otherwise successful trajectories, and rerun Traj-Bootstrap on ALFWorld with the same budget. If success rates stay near the clean 89%, the method's benefit does not depend on filtering by verified success; if they drop toward the 73% Fixed-DB baseline, the binary success signal is doing the load-bearing work the paper claims.","tokens_in":19456,"feed_emoji":"🤖","tokens_out":7135,"duration_ms":71014,"temperature":0.7,"pith_summary":"The paper asks whether an LLM-based agent can improve on new tasks purely by remembering its own past successes, without task-specific prompts or hand-built action spaces. Its claim is yes: storing every successfully completed trajectory in a retrieval database and feeding those trajectories back as in-context examples raises success rates from 73% to 89% on ALFWorld, from 55% to 64% on Wordcraft, and from 75% to 79% on InterCode-SQL. Two additional curation stages, selecting good whole databases through population-based training and pruning individual trajectories by measured usefulness, push ALFWorld to 93%. The broader point is that a data-centric loop of self-collected experience can produce gains comparable to or larger than upgrading the underlying model, while requiring only a binary success signal from the environment. Agent improvement, on this view, does not have to come from human knowledge engineering.","feed_headline":"LLM agents gain up to 20 points by bootstrapping their own wins","feed_subtitle":"Saving successful trajectories as examples beats a model upgrade on three decision benchmarks.","key_machinery":"The load-bearing object is the trajectory database $D$, whose entries are full episodes of the form (goal, plan, observations, reasoning, actions, success bit). The agent retrieves from $D$ at each decision point by average cosine similarity over multiple keys, goal and plan for the initial plan and then observation or reasoning for each step, and feeds the retrieved segments into the LLM as in-context examples. Growth is a positive feedback loop: only successful trajectories ($s=1$) are stored, and they help solve new tasks, generating more successful trajectories. Two curation mechanisms sit on top: +DB-Curation runs population-based training over several database instances, periodically replacing poor-performing databases with copies of the best, and +Exemplar-Curation scores each trajectory by $Q(\\tau)$, the retrieval-frequency-weighted average outcome of the tasks where it was used, keeping the top exemplar per task. This retrieval-weighted quality score is what lets the method distinguish trajectories that are genuinely instructive from those that merely happened to end in success.","core_discovery":"The paper's central claim is that an agent's own successful experience is a reusable resource: trajectories that ended in reward, when stored in a database and retrieved at each planning, reasoning, and acting step, make the agent generalize better to held-out tasks. The evidence is that this holds across three benchmarks without changing the prompt templates, retrieval mechanism, or embeddings. The paper also claims that not all successful trajectories are equally instructive, because some succeed despite containing bad decisions; a retrieval-weighted quality score that tracks how often a trajectory is used in tasks that end successfully identifies the more valuable exemplars. Combining whole-database selection with exemplar-level pruning reaches 93% on ALFWorld, a result the authors interpret as surpassing approaches that combine more powerful LLMs with hand-crafted observation and action spaces.","pith_inferences":["A testable extension: the method should transfer to other long-horizon agentic settings, such as browser automation or instruction-following robots, whenever the environment supplies a clean binary success signal, because the prompts, embeddings, and retrieval are deliberately task-agnostic; I would expect the largest gains where the base agent is weakest at exploration.","The best-versus-worst exemplar comparison also suggests a cheap data-filtering heuristic for other training pipelines: rank trajectories by downstream utility rather than by raw success, which could curate data for supervised fine-tuning without human labels.","Because the method leans on a clean success bit, a natural stress test is whether performance degrades gracefully under noisy or partial rewards; if it does not, coupling trajectory bootstrapping with a learned verifier would be a natural next step.","The success-prediction result hints that difficulty estimation and example selection could be coupled: retrieve examples preferentially from tasks predicted hard, rather than by similarity alone."],"forward_implications":["On all three benchmarks, a single-attempt agent using Traj-Bootstrap matches or beats the Fixed-DB baseline given two or three independent attempts, so bootstrapping can substitute for test-time sampling when repeated attempts are impractical.","The performance gain from database construction (20 points on ALFWorld after 3,500 tasks) exceeds the 15-point gain from upgrading the underlying LLM from gpt-4o-mini to gpt-4o.","Combined database- and exemplar-level curation with gpt-4o-mini reaches 93% on ALFWorld, above the 91% reported for AutoManual, which uses gpt-4-turbo plus gpt-4o-mini and hand-crafted observation and action spaces.","The same self-collected trajectories can be used to fine-tune the base model, producing agents competitive with the in-context versions (ALFWorld 96% vs 93%, Wordcraft 74% vs 69%), so the database is a reusable asset rather than a prompt-only artifact.","As the database grows, a classifier trained on goal and initial observation embeddings predicts task success with improving AUROC (0.77 on InterCode-SQL and 0.71 on Wordcraft), so the collected experience also supports diagnostics and task-difficulty estimates."],"supporting_citations":[{"why":"Supplies the ReAct-style reasoning-and-acting loop that the agent builds on.","marker":"[9]"},{"why":"Introduces the retrieval-augmented planning and per-step retrieval scheme the agent adapts.","marker":"[10]"},{"why":"Provides the initial human-written exemplars and the prior experiential-learning baseline (ExpeL).","marker":"[17]"},{"why":"Gives the reward-weighted regression principle that justifies storing only successful trajectories.","marker":"[33]"},{"why":"Supplies population-based training as the mechanism for database-level curation.","marker":"[34]"},{"why":"Is the hand-crafted AutoManual baseline that the combined method outperforms on ALFWorld.","marker":"[4]"},{"why":"Is the hierarchical rule-learning AutoGuide baseline compared at equal training-task counts.","marker":"[12]"},{"why":"Defines the ALFWorld benchmark used for the main success-rate results.","marker":"[36]"},{"why":"Defines the InterCode-SQL benchmark and the hand-crafted GameSQL comparison agent.","marker":"[37]"},{"why":"Defines the Wordcraft benchmark used for the compositional reasoning evaluation.","marker":"[38]"}],"fun_headline_variants":["Self-taught LLM agents beat model upgrade by 20 points","Agents learn from own wins, outpace model upgrades","Storing own successful actions lifts LLM agents","LLM agents improve by reusing their own successes","Self-bootstrapping beats GPU upgrade for LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The environment must supply a reliable binary success signal at the end of each episode, and the agent only stores episodes marked successful; if that signal is wrong or missing, failed trajectories fill the database and the improvement is expected to vanish.","fun_headline_variants_meta":{"raw":{"variants":["Self-taught LLM agents beat model upgrade by 20 points","Agents learn from own wins, outpace model upgrades","Storing own successful actions lifts LLM agents","LLM agents improve by reusing their own successes","Self-bootstrapping beats GPU upgrade for LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2845,"prompt_tokens":930,"completion_tokens":1915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1834}},"tokens_in":546,"tokens_out":1915,"duration_ms":12471,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:47:24.919671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt the success signal during training, for example by flipping the stored success bit for 20% of otherwise successful trajectories, and rerun Traj-Bootstrap on ALFWorld with the same budget. If success rates stay near the clean 89%, the method's benefit does not depend on filtering by verified success; if they drop toward the 73% Fixed-DB baseline, the binary success signal is doing the load-bearing work the paper claims.","supporting_citations":[{"cited_title":"WordCraft: An Environment for Benchmarking Commonsense Agents","cited_arxiv_id":"2007.09185","evidence_quote":"Defines the Wordcraft benchmark used for the compositional reasoning evaluation."}],"review_version":1}