Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that LLM agents can improve on new sequential-decision tasks by accumulating their own successful trajectories and using them as in-context examples.

desk verdict A clean empirical study of bootstrapping retrieval databases, but the missing control leaves the 'self-generated' claim underdetermined. read the letter →

arxiv 2505.00234 v3 pith:62X4MDFD submitted 2025-05-01 cs.LG cs.CL

Vishnu Sarukkai , Zhiqiang Xie , Kayvon Fatahalian This is my paper · ORCID
classification cs.LGcs.CL
keywords LLMagentsin-contextlearningtrajectorybootstrappingsequentialdecision-makingretrieval-augmentedself-improvementdatacurationpopulation-basedtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether an LLM-based agent can improve on new tasks purely by remembering its own past successes, without task-specific prompts or hand-built action spaces. Its claim is yes: storing every successfully completed trajectory in a retrieval database and feeding those trajectories back as in-context examples raises success rates from 73% to 89% on ALFWorld, from 55% to 64% on Wordcraft, and from 75% to 79% on InterCode-SQL. Two additional curation stages, selecting good whole databases through population-based training and pruning individual trajectories by measured usefulness, push ALFWorld to 93%. The broader point is that a data-centric loop of self-collected experience can produce gains comparable to or larger than upgrading the underlying model, while requiring only a binary success signal from the environment. Agent improvement, on this view, does not have to come from human knowledge engineering.

What carries the argument

The load-bearing object is the trajectory database $D$, whose entries are full episodes of the form (goal, plan, observations, reasoning, actions, success bit). The agent retrieves from $D$ at each decision point by average cosine similarity over multiple keys, goal and plan for the initial plan and then observation or reasoning for each step, and feeds the retrieved segments into the LLM as in-context examples. Growth is a positive feedback loop: only successful trajectories ($s=1$) are stored, and they help solve new tasks, generating more successful trajectories. Two curation mechanisms sit on top: +DB-Curation runs population-based training over several database instances, periodically replacing poor-performing databases with copies of the best, and +Exemplar-Curation scores each trajectory by $Q(\tau)$, the retrieval-frequency-weighted average outcome of the tasks where it was used, keeping the top exemplar per task. This retrieval-weighted quality score is what lets the method distinguish trajectories that are genuinely instructive from those that merely happened to end in success.

What would settle it

Corrupt the success signal during training, for example by flipping the stored success bit for 20% of otherwise successful trajectories, and rerun Traj-Bootstrap on ALFWorld with the same budget. If success rates stay near the clean 89%, the method's benefit does not depend on filtering by verified success; if they drop toward the 73% Fixed-DB baseline, the binary success signal is doing the load-bearing work the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that an agent's own successful experience is a reusable resource: trajectories that ended in reward, when stored in a database and retrieved at each planning, reasoning, and acting step, make the agent generalize better to held-out tasks. The evidence is that this holds across three benchmarks without changing the prompt templates, retrieval mechanism, or embeddings. The paper also claims that not all successful trajectories are equally instructive, because some succeed despite containing bad decisions; a retrieval-weighted quality score that tracks how often a trajectory is used in tasks that end successfully identifies the more valuable exemplars. Combining whole-database selection with exemplar-level pruning reaches 93% on ALFWorld, a result the authors interpret as surpassing approaches that combine more powerful LLMs with hand-crafted observation and action spaces.

Load-bearing premise

The environment must supply a reliable binary success signal at the end of each episode, and the agent only stores episodes marked successful; if that signal is wrong or missing, failed trajectories fill the database and the improvement is expected to vanish.

Editorial extensions

If this is right

  • On all three benchmarks, a single-attempt agent using Traj-Bootstrap matches or beats the Fixed-DB baseline given two or three independent attempts, so bootstrapping can substitute for test-time sampling when repeated attempts are impractical.
  • The performance gain from database construction (20 points on ALFWorld after 3,500 tasks) exceeds the 15-point gain from upgrading the underlying LLM from gpt-4o-mini to gpt-4o.
  • Combined database- and exemplar-level curation with gpt-4o-mini reaches 93% on ALFWorld, above the 91% reported for AutoManual, which uses gpt-4-turbo plus gpt-4o-mini and hand-crafted observation and action spaces.
  • The same self-collected trajectories can be used to fine-tune the base model, producing agents competitive with the in-context versions (ALFWorld 96% vs 93%, Wordcraft 74% vs 69%), so the database is a reusable asset rather than a prompt-only artifact.
  • As the database grows, a classifier trained on goal and initial observation embeddings predicts task success with improving AUROC (0.77 on InterCode-SQL and 0.71 on Wordcraft), so the collected experience also supports diagnostics and task-difficulty estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the method should transfer to other long-horizon agentic settings, such as browser automation or instruction-following robots, whenever the environment supplies a clean binary success signal, because the prompts, embeddings, and retrieval are deliberately task-agnostic; I would expect the largest gains where the base agent is weakest at exploration.
  • The best-versus-worst exemplar comparison also suggests a cheap data-filtering heuristic for other training pipelines: rank trajectories by downstream utility rather than by raw success, which could curate data for supervised fine-tuning without human labels.
  • Because the method leans on a clean success bit, a natural stress test is whether performance degrades gracefully under noisy or partial rewards; if it does not, coupling trajectory bootstrapping with a learned verifier would be a natural next step.
  • The success-prediction result hints that difficulty estimation and example selection could be coupled: retrieve examples preferentially from tasks predicted hard, rather than by similarity alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes trajectory bootstrapping as a way to build retrieval databases of in-context examples for ReAct-style LLM agents in sequential decision-making tasks. Starting from a small set of human-provided exemplars, Traj-Bootstrap appends only successful self-generated training trajectories to the database, which is queried at each decision point with a multi-key embedding retrieval scheme. On three benchmarks (ALFWorld, Wordcraft, InterCode-SQL), naive accumulation raises held-out success rates from 73% to 89%, from 55% to 64%, and from 75% to 79%, respectively. Two curation mechanisms are then introduced: population-based database-level curation (+DB-Curation) and exemplar-level curation (+Exemplar-Curation) based on a retrieval-weighted quality metric Q(τ). The combined method reaches 93% on ALFWorld and 82% on InterCode-SQL. The paper contextualizes the gains against test-time sampling (pass@k), LLM upgrades, fine-tuning on the collected data, and prior systems (AutoManual, Autoguide).

Significance. If the central claim is established, the paper contributes a practically appealing, task-agnostic mechanism for agent self-improvement that avoids prompt engineering, and the reported gains (e.g., matching pass@2–pass@3 under a single test-time attempt) are meaningful. The manuscript has genuine strengths: evaluations use five seeds with clean train/test separation (App. F.2); the pass@k comparison in Table 3 is a fair and interpretable baseline; Fig. 4's best-versus-worst exemplar comparison is a good causal check on the curation metric; and App. E directly tests whether the method can bootstrap from an empty database. The fine-tuning results in App. C indicate the collected data have independent value. However, the central causal claim that self-generation, rather than database size or the success filter, drives the gains is not isolated by the experiments; the paper's own invocation of many-shot in-context learning results (refs. [8, 22]) highlights this gap. The headline contextual claims in the abstract (exceeding model upgrades, surpassing AutoManual) are also stronger than the measurements support.

major comments (3)
  1. [§5.1, §6.2, Table 1] The central claim that self-generated trajectories are the active mechanism is underdetermined by the experiments. The only comparison offered is Fixed-DB (18 hand-written trajectories on ALFWorld, per §H.1) versus Traj-Bootstrap (up to 3,500 agent-generated successful trajectories), which jointly changes database size and diversity, the success filter of §5.1, and the source of the trajectories. No control populates the database with (a) all trajectories including failures, or (b) successful trajectories from an external, non-bootstrapping policy (for example, a stronger LLM or an independent agent). Without (b), the gains are consistent with the known many-shot in-context-learning effect that the authors themselves invoke in §6.2 (refs. [8, 22]); without (a), the contribution of the success filter is never isolated. Appendix E shows that bootstrapping from an empty database still scales with database size on Wordcraft, which is partial evidence for self-generation, but it still does not compare against an external source of successful trajectories. Because the title and abstract make the self-generated property the mechanism, this missing source control is load-bearing; I recommend adding both controls on at least ALFWorld and Wordcraft and reporting whether the Traj-Bootstrap gains survive.
  2. [§6.3, Table 2, Abstract] The abstract claims that the reported improvements 'exceed those achieved by upgrading from gpt-4o-mini to gpt-4o,' but this comparison is shown only for ALFWorld (Table 2), and even there the difference is not statistically meaningful: Traj-Bootstrap improves 0.73 to 0.89 (16 points) while the model upgrade improves 0.73 to 0.88 (15 points), with overlapping standard errors. The §6.3 sentence stating that the 20-point boost 'significantly outperforms' the 15-point upgrade is not supported by any significance test, and the 20-point figure itself appears to be an arithmetic error (0.73 to 0.89 is 16 points; 0.73 to 0.93 for the combined method is 20 points). The abstract and §6.3 should be revised to state the ALFWorld-only comparison with appropriate uncertainty.
  3. [§6.3, Table 2, Abstract] The 'surpassing' comparison with AutoManual in §6.3 and the abstract uses very different training budgets: +DB+Exemplar-Curation is trained on 3,500 tasks (with five parallel instances under +DB-Curation), while the AutoManual rows in Table 2 use 36 tasks. As reported, the comparison bundles algorithmic merit with a roughly two-order-of-magnitude difference in environment interactions, so 'surpassing approaches that use more powerful LLMs and hand-crafted components' overstates what is established. Please report the number of LLM calls (the paper gives totals only in App. I, not per method) and either match the baseline's training budget or evaluate both methods at 36, 100, and 3,500 tasks so the scaling behavior is visible.
minor comments (6)
  1. [Abstract, §3] The abstract's 'without human intervention' overstates what the method requires: §3 assumes an initial set of human-generated trajectories and a descriptor of the action space, and Appendix E shows that removing the human examples permanently lowers Wordcraft performance even after 4,000 collected trajectories; please qualify this phrase.
  2. [§2.1, §5.1] The method relies on a reliable binary success signal at training time, which is stated in §2.1 and used as the storage filter in §5.1; the paper should explicitly discuss robustness to noisy or unavailable success signals, since contamination of the database by failed trajectories would likely remove the reported benefit.
  3. [§5.3, B.4] Equation (2) in Appendix B.4 restates Equation (1) from §5.3; please cross-reference instead of redefining the metric, and decide on a single numbering.
  4. [§H.1, §H.2] The benchmark-specific choices (category-based similarity on ALFWorld; appending the solution query to the goal on InterCode-SQL) soften the claim of avoiding task-specific engineering; clarify whether these follow the prior conventions of [10] and [37], and test whether the InterCode-SQL result persists without the goal-append modification.
  5. [§6.3, Table 3] Please state the sampling temperature used for the Fixed-DB pass@k attempts in Table 3; the k-attempt equivalence is only meaningful if the sampling distribution matches the temperature (0.1) used for the single-attempt evaluations.
  6. [Various] There are several typographical errors, including 'Traj-Boostrap' in the Table 2 caption and 'appraoches' in Appendix A, and the benchmark name appears both as 'Intercode-SQL' and 'InterCode-SQL'; please normalize these.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: evaluation is on held-out tasks and the paper contains no load-bearing self-citations.

full rationale

None of the paper's claims reduces to its inputs by construction. The database is built by storing only trajectories with environment-verified success (s = 1) on a designated training split, and the reported success rates are measured on held-out test tasks (e.g., ALFWorld 134 out-of-distribution test tasks, InterCode-SQL 234 test tasks, Wordcraft 500 test tasks). Consequently, the central improvement from Traj-Bootstrap cannot be an identity between fitted data and evaluation. The Q metric in Eq. 1 is an in-training validation statistic used to select exemplars or whole databases, but the final numbers come from applying the resulting fixed database to a separate test distribution; this is standard model selection rather than a self-prediction. There are no load-bearing self-citations: the reference list contains no papers by Sarukkai/Xie/Fatahalian, and the cited prior results (ReAct, RAP, ExpeL, AutoGuide, AutoManual, etc.) are used as baselines or component design choices, not as the source of the claimed improvement. The paper also explicitly states in Appendix F.2 that trajectories used for database construction come exclusively from training tasks, with evaluation on separate test tasks never seen during construction. The strongest legitimate concern is an experimental confound: Traj-Bootstrap differs from Fixed-DB in both database size and trajectory source, so the 'self-generated' attribute is not isolated by the main comparison. However, underdetermination by missing controls is an experimental design issue, not a circular derivation, and the paper does provide an additional empty-database ablation in Appendix E. Therefore, no circular step meets the evidentiary bar, and the paper is self-contained against external benchmarks.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The ledger is mostly experimental hyperparameters and domain assumptions; no new physical or mathematical entities are introduced. The method's core precondition is the existence of a verifiable success signal in the training environment.

free parameters (7)
  • retrieval top-k = 6 (ALFWorld, InterCode-SQL), 10 (Wordcraft)
    Number of in-context trajectories retrieved per decision step; set per benchmark without a reported sweep.
  • population size N = 5
    Number of parallel database instances in +DB-Curation; chosen without ablation.
  • curation schedule = evaluate at 10, 20, 40, 80, ... tasks
    Doubling threshold for triggering database replacement in +DB-Curation.
  • state window size = 5 states
    Sliding window around the most similar state in retrieved trajectories.
  • minimum retrieval count for Q metric = 3 tasks
    Threshold below which trajectories get a neutral score equal to the average success rate.
  • LLM temperature = 0.1
    Sampling temperature for all LLM calls; not swept.
  • number of training tasks = 3500 (ALFWorld), 800 (InterCode-SQL), 4000 (Wordcraft)
    These sizes determine the final database size and reported success rates.
assumptions (5)
  • domain assumption Environments provide a reliable binary success signal at episode end.
    Stated in Sec. 2.1 as sparse reward R=1/0 and used in Sec. 5.1 to filter trajectories.
  • domain assumption The frozen embedding model all-MiniLM-L6-v2 captures semantic similarity for retrieval.
    Used for all retrieval keys in Algorithm 4; no task-specific embedding is used.
  • domain assumption A ReAct-style agent with planning and per-step retrieval is an appropriate base architecture.
    Assumed as the foundation; all comparisons are relative to this base.
  • domain assumption The train and test splits are representative of the task distribution.
    Evaluation uses fixed splits; ALFWorld test is OOD from training and Wordcraft goals are disjoint.
  • domain assumption Five random seeds are sufficient to estimate success-rate variance.
    All main results average over 5 trials; no significance tests are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks." pith.science (2026). https://pith.science/paper/62X4MDFD

@misc{pith2026250500234,
  author       = {Pith},
  title        = {Pith review of: Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/62X4MDFD}},
  note         = {Machine review of arXiv:2505.00234}
}
read the original abstract

Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering--custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73% to 89%), Wordcraft (55% to 64%), and InterCode-SQL (75% to 79%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93% success on ALFWorld--surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.

Figures

Figures reproduced from arXiv: 2505.00234 by the authors.

Figure 1
Figure 1. Traj-Bootstrap leads to variance in test-time suc￾cess rate. Individual trials (5) shown as dashed lines, results on Intercode-SQL benchmark. There is noticeable variability in performance across trials. The Traj-Bootstrap algorithm displays unpredictable per￾formance variation across training trials, even when fol￾lowing identical collection procedures [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Traj-Bootstrap results: success rate improves with increasing training tasks on all three benchmarks. Individual trials (5) shown as dashed lines. All benchmarks exhibit diminishing returns as the database size increases. Trials show substantial performance variability, both within individual trials and across different trials. +DB-Curation boosts performance on ALFWorld [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Success rate comparison for Traj-Bootstrap and its variants (+DB-Curation, +Exemplar-Curation, +DB+Exemplar-Curation). +DB-Curation enhances final success rate only on ALFWorld, but improves success rate for smaller DB sizes on all benchmarks. +Exemplar-Curation delivers success rate gains on both Intercode-SQL and Wordcraft. The combination of both enhancements delivers the largest gains on both ALFWorld and InterC… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The ‘best’ bootstrapped trajectories compared to the ‘worst’. Databases con￾structed from the highest-quality successful trajectory per task, as measured by Eq. 1, outperform databases built from the lowest-quality successful trajectories on both ALF￾World and Wordcraf…
Figure 5
Figure 5. Figure 5: AUROC of success prediction improves with more self-collected examples. Performance continues to rise with increasing database size. Tab. 4 shows that ReAct-Finetune slightly outperforms the in-context agent on ALFWorld (0.96 vs. 0.93) and Wordcraft (0.74 vs. 0.69), wh…
Figure 6
Figure 6. Figure 6: Predicted probabilities are well-calibrated. For both benchmarks, predicted and empirical success rates generally align. 0 1000 2000 3000 4000 Num. Training Tasks 0.45 0.50 0.55 0.60 Success Rate Wordcraft Traj-Bootstrap -Human_Examples [PITH_FULL_IMAGE:figures/full_f…
Figure 7
Figure 7. Figure 7: Ablating the value of initial human-provided examples, Wordcraft. Traj￾Bootstrap, initialized by default with a database of 5 human-provided trajectories for Wordcraft, achieves better performance with these starting examples than when initialized from an empty databas…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 5 canonical work pages

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  2. [2]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Larger language models do in-context learning differently

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023

    arXiv 2023

  4. [4]

    Au- tomanual: Generating instruction manuals by llm agents via interactive environmental learning

    Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. Au- tomanual: Generating instruction manuals by llm agents via interactive environmental learning. arXiv preprint arXiv:2405.16247, 2024

    arXiv 2024

  5. [5]

    Agentoccam: A simple yet strong baseline for llm- based web agents

    Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm- based web agents. arXiv preprint arXiv:2410.13825, 2024

    arXiv 2024

  6. [6]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky ¨urek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022

    arXiv 2022

  7. [7]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo˜ao Sacramento, Alexan- der Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR, 2023

    work page 2023

  8. [8]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930–76966, 2024

    2024

Show all 43 references
  1. [9]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  2. [10]

    Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents

    Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents. arXiv preprint arXiv:2402.03610, 2024

    2024 arXiv

  3. [11]

    Trad: Enhancing llm agents with step-wise thought retrieval and aligned decision

    Ruiwen Zhou, Yingxuan Yang, Muning Wen, Ying Wen, Wenhao Wang, Chunling Xi, Guoqiang Xu, Yong Yu, and Weinan Zhang. Trad: Enhancing llm agents with step-wise thought retrieval and aligned decision. In Proceedings of the 47th International ACM SIGIR Conference on Research and D...

    2024

  4. [12]

    Autoguide: Automated generation and selec- tion of context-aware guidelines for large language model agents

    Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selec- tion of context-aware guidelines for large language model agents. arXiv preprint arXiv:2403.08978, 2024

    2024 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  6. [14]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  7. [15]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009, 2023

    2023

  8. [16]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  9. [17]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    2024

  10. [18]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  11. [19]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  12. [20]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  13. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  14. [22]

    In-context learning with long-context models: An in-depth exploration

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Gra- ham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024

    2024 arXiv

  15. [23]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav San- thanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    arXiv

  16. [24]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695, 2024

    2024 arXiv

  17. [25]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    2024 arXiv

  18. [26]

    Planning in natural language improves llm search for code generation

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733, 2024. 13

    2024 arXiv

  19. [27]

    Scaling inference-time search with vision value model for improved visual comprehension

    Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704, 2024

    2024 arXiv

  20. [28]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  21. [29]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  22. [30]

    Archon: An architecture search framework for inference-time techniques

    Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher R ´e, et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254, 2024

    2024 arXiv

  23. [31]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024

    2024 arXiv

  24. [32]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024

    2024 arXiv

  25. [33]

    Reinforcement learning by reward-weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  26. [34]

    Population based training of neural networks

    Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Don- ahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017

    2017 arXiv

  27. [35]

    Reinforcement learning: An introduction

    Andrew G Barto. Reinforcement learning: An introduction. by richard’s sutton. SIAM Rev, 6(2):423, 2021

    2021

  28. [36]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆot´e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020

    2010 arXiv

  29. [37]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

    2023

  30. [38]

    Wordcraft: An environment for benchmarking commonsense agents

    Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip HS Torr, Shimon Whiteson, and Tim Rockt¨aschel. Wordcraft: An environment for benchmarking commonsense agents. arXiv preprint arXiv:2007.09185, 2020

    2007 arXiv

  31. [39]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http: //arxiv.org/abs/1908.10084

    2019 arXiv

  32. [40]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar´e, Maria Lomeli, Lucas Hosseini, and Herv´e J´egou. The faiss library. arXiv preprint arXiv:2401.08281, 2024

    2024 arXiv

  33. [41]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  34. [42]

    goal,plan

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explain- able multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. 14 A Key Agent Details In Sec. 2, we estab...

    2018 arXiv

  35. [43]

    Find the first name of a student who have both cat and dog pets

    since Wordcraft is a slightly simpler task, requiring up to 4 steps for task completion while InterCode-SQL requires up to 10. We used the following action space string for Wordcraft: Output strings with the names of the two entities we would like to combine in this step. H.4 ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.