Pith. sign in

REVIEW 4 major objections 5 minor 39 references

SAND: Boosting LLM Agents with Self-Taught Action Deliberation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that finetuning an LLM agent on self-synthesized action deliberation trajectories improves task success on ALFWorld and ScienceWorld by about 20% over supervised finetuning, without new human labels.

desk verdict A credible self-training recipe for LLM agents with a believable 20% gain, but the forced-conclusion deliberation and the disabled expert-switch expose an untested optimality assumption that deserves referee scrutiny. read the letter →

arxiv 2507.07441 v2 pith:YOCFBCT4 submitted 2025-07-10 cs.CL

Yu Xia , Yiran Shen , Junda Wu , Tong Yu , Sungchul Kim , Ryan A. Rossi , Lina Yao , Julian McAuley This is my paper
classification cs.CL
keywords LLMagentsactiondeliberationself-trainingiterativefinetuningself-consistencysamplingexecution-guidedcritiqueALFWorldScienceWorld
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM agent can make better decisions in interactive environments if, during training, it is forced to deliberate over alternative actions instead of merely imitating an expert's choice. The proposed SAND (Self-taught ActioN Deliberation) framework starts from a supervised-finetuned ReAct-style agent, then repeatedly uses the base model itself to synthesize step-level deliberation thoughts: sample several candidate actions, roll each one out in the environment, critique the outcomes, and write a short comparison that ends by justifying the expert action. These enriched trajectories become the new training data, and the loop repeats. The central empirical claim is that on ALFWorld and ScienceWorld, SAND improves average reward by roughly 20% over initial supervised finetuning and outperforms existing agent-tuning baselines, with the gains concentrated on unseen tasks. If correct, this matters because it offers a path to stronger interactive agents using only the model's own synthetic reasoning and environment feedback, with no additional human labels.

What carries the argument

The load-bearing object is the synthesized deliberation trajectory, which is an expert trajectory with additional deliberation thoughts inserted at selected steps. Selection is driven by a self-consistency inconsistency indicator: at each expert step the current policy samples $N$ candidate actions, and if the set $\{\hat a_t^{(1)},\dots,\hat a_t^{(N)}, a_t\}$ contains more than one unique action, the step is flagged for deliberation. For each flagged step, SAND rolls out every candidate action in the environment to obtain a reward, asks the frozen base LLM to write a verbal critique of each action using that rollout, and finally prompts the base LLM to synthesize one thought that compares the candidates and concludes with the expert action. The agent is then finetuned on these enriched trajectories, and the loop repeats. The key design choice is that the critiques are execution-grounded and the deliberation is forced to be consistent with the expert choice, so the model learns a rationalization anchored to actual outcomes rather than to free-form speculation.

What would settle it

Take a held-out subset of the expert trajectories, replace every flagged expert action with a deliberately bad alternative (for example the sampled candidate with the lowest rollout reward), and run the full SAND loop forcing the deliberation to conclude with that bad action. If the finetuned agent still matches SAND's reported gains on unseen tasks, the expert-optimality assumption is not load-bearing; if it adopts the bad actions and performance collapses relative to the unmodified SAND agent, the assumption is confirmed as a necessary condition for the result.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an LLM agent tuned on deliberation trajectories—where each synthesized thought lists candidate actions, evaluates them with execution-informed critiques, and explains why the expert action wins—learns both what to do and when to think longer. Experimentally, after three iterations on Llama-3.1-8B-Instruct the average reward rises from 72.9 with SFT to 88.9, and on Qwen2.5-7B-Instruct from 69.4 to 84.6, across seen and unseen ALFWorld and ScienceWorld tasks; this is the average 20% improvement over initial supervised finetuning reported in the abstract. The finetuned agent does not sample at inference time: it emits the full deliberation thought and the chosen action in one pass, yet it generalizes to unseen tasks better than the SFT agent and better than strong baselines such as ETO, WKM, and MPO. The authors also report that the learned deliberation rate is higher on hard tasks and lower on easy ones, indicating the agents acquire a sense of when deliberation is worth the extra tokens.

Load-bearing premise

SAND assumes the expert action in each training trajectory is at least as good as any sampled alternative, because the synthesized deliberation is forced to conclude with that action; the paper states this assumption explicitly in Section 4.4, and its own expert-switch mechanism—disabled on ScienceWorld—is the admitted countermeasure when the assumption fails.

Editorial extensions

If this is right

  • Supervised finetuning on expert trajectories leaves a failure mode—over-committing to plausible but wrong actions—that explicit action deliberation during training directly addresses.
  • Across three iterations, SAND steadily improves average reward on both backbone models, so the self-training loop is the source of the gains rather than a one-time data augmentation.
  • The finetuned agent learns when to deliberate: measured deliberation rates are higher on hard ScienceWorld tasks and lower on easy ones, so the extra reasoning is spent where it pays.
  • Inference stays single-pass and sampling-free at test time, with roughly 2 to 3 times the tokens per task of SFT—below the 5 times of a Best-of-N baseline.
  • The method is complementary to test-time search methods guided by process reward or Q-value models, so training-time deliberation can be combined with test-time search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because SAND forces every synthesized deliberation to end with the expert action, the training signal inherits whatever suboptimality is in the expert trajectories; the authors' own decision to disable the expert-switch mechanism on ScienceWorld hints that this assumption is not always safe, and a variant that sometimes argues for a better explored alternative might generalize further.
  • The candidate actions are sampled from the current policy, so the diversity of deliberation depends on the policy's own uncertainty; after several iterations the policy may become more peaked, shrinking candidate sets and possibly capping further gains, so a natural stopping criterion might be when the inconsistency flag rarely fires.
  • Execution-guided critique requires an environment simulator or rollout oracle; extending SAND to domains without cheap rollouts (for example real web transactions or dialogue) would require a learned outcome model or human feedback to supply the critiques.
  • A cost-benefit comparison beyond Best-of-N—for example against a fixed deliberation budget or against prompting-based deliberation without finetuning—would sharpen the paper's claim that 2 to 3 times token overhead is a reasonable price for the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SAND, an iterative self-training framework for LLM agents. Starting from an SFT policy on ReAct-style expert trajectories, SAND samples candidate actions via self-consistency, executes each candidate to obtain rollout rewards, uses a frozen base LLM to generate execution-guided critiques, and then prompts the same base LLM to synthesize a deliberation thought that is forced to conclude with the expert action. These augmented 'deliberation trajectories' are used to finetune the agent, and the process repeats for three iterations. Experiments on ALFWorld, ScienceWorld, and WebShop report an average improvement of roughly 20% over the initial SFT agent and better results than several agent-tuning baselines.

Significance. If the empirical results hold, SAND is a practical and relatively simple recipe for improving open-weight LLM agents on interactive text environments without additional human labels. The framework is clearly presented, and the two main ablations (removing self-consistency sampling or removing execution-guided critiques) go in the expected direction, which strengthens the internal consistency of the story. The WebShop appendix also provides some evidence of generality to a third environment. However, the absence of statistical controls, the forced-conclusion deliberation prompt, and the use of imported baseline numbers mean that the size and the mechanism of the reported gains are not yet established at the level the abstract claims.

major comments (4)
  1. [Section 5.1, Table 2] The headline results are reported without error bars, multiple seeds, or significance tests, and several comparisons (explicitly in Appendix B, Table 5) are 'directly imported from the original papers.' Because the central claim is an empirical superiority claim over SFT and other tuning methods, the absence of variance estimates and rerun baselines makes the 20% improvement and 'outperforms state-of-the-art' statements unsupported by evidence. I request at least 3-5 seeds with standard deviations and, where feasible, a rerun of the closest baselines (ETO, WKM, MPO) under the same data, prompts, and training setup.
  2. [Section 4.4 and Appendix C (Figure 6)] The deliberation synthesis prompt instructs the model: 'Your final Action line must be {expert_action}. Everything you write has to lead naturally to this choice.' Combined with the assumption in Section 4.4 that the expert action is optimal, this means every synthesized deliberation is a post-hoc rationalization of the expert choice, regardless of what the execution-guided critiques indicate. The expert-switch mechanism is an admission that expert actions can be suboptimal, yet it is disabled on ScienceWorld (Section 5.3). The paper provides no analysis of how often the forced conclusion contradicts the critiques or how such contradictions affect the learned policy. This is load-bearing for the claim that SAND teaches genuine deliberation; please add an analysis of deliberation faithfulness (e.g., compare the expert action to the highest-reward rollout, or ablate the forced-conclusion instruction).
  3. [Sections 5.3 and 6.5] SAND uses 2-3x more inference tokens than the SFT agent (Table 4) and trains for additional epochs on synthesized trajectories. There is no token-matched or compute-matched control, so a reader cannot tell whether the gains come from the deliberation content or simply from more inference-time computation and more training data. I recommend adding an SFT baseline trained on the same number of augmented trajectories with the deliberation thoughts removed (or with additional expert trajectories), and a Best-of-N control with a matched token budget.
  4. [Sections 4.3 and 5.3] The decision to disable the expert-switch mechanism on ScienceWorld because 'some of the tasks have short-cuts that might boost LLM agents on training set but hurt performances on test set' indicates that the rollout reward used to generate execution-guided critiques is not aligned with test success in that environment. Since the critiques are a key training signal, the paper should report how often the rollout reward and the final test reward disagree, or otherwise justify that the critiques remain informative despite this misalignment.
minor comments (5)
  1. [Algorithm 1] The condition 'if |{â(1)t, ..., â(N)t, at}| = 0 then continue' is never true because the expert action at is always in the set; it should be '= 1' to match Eq. (4).
  2. [Section 4.5] The reference to 'Equation 4.2' should be to Eq. (4).
  3. [Throughout] The typography for the method name is inconsistent: 'SA ND', 'SAnD', and 'SA ND' appear in different places; please unify to SAND.
  4. [Figure 3 caption] The caption contains 'Average Deliberat Rate'; this should be 'Average Deliberation Rate'.
  5. [Section 6.2] The statement that 'SANDw/o SAS can even hurt the agent performance being outperformed by initial SFT' is only true for ScienceWorld in Table 3; on ALFWorld the variant still improves over SFT. The text should qualify this.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAND's central claim is an external held-out benchmark result, and the forced-conclusion deliberation prompt is an explicit optimality assumption rather than a circular derivation.

full rationale

The paper's central claim is empirical: SAND improves average reward on held-out test tasks in ALFWorld, ScienceWorld, and WebShop relative to SFT and other baselines (Tables 2, 5, and 6). These results are evaluated against external environments and are not derived from the method's own inputs by construction. The closest thing to a circular step is the action deliberation synthesis prompt in Appendix C, which requires the synthesized thought's final action to be the expert action ('Your final Action line must be {expert_action}. Everything you write has to lead naturally to this choice.'). However, the paper explicitly discloses the underlying assumption in Section 4.4: 'we keep the expert action at as the ground-truth action here assuming it is the optimal one at the current step.' This is a stated modeling assumption about the quality of expert data, not a hidden equivalence between an input and an output. The self-training nature of SAND—using the base model to generate deliberation trajectories and then fine-tuning on them—does not make the held-out improvement circular, because the test-time evaluation is independent of the training-data generation process. The paper's self-citations, including the Table 5 comparison numbers imported from Xia et al. (2025a), are used as baseline results or background context, not as load-bearing justification for the central improvement claim. No equation in the paper reduces to its own inputs, and no fitted parameter is renamed as a prediction. The identified weakness—that expert actions may be suboptimal or non-unique—is a correctness-risk concern about the method's assumption, not a circularity in the derivation or evaluation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on expert-trajectory quality, the reliability of environment rewards for step-level critique, the coverage of N-sample self-consistency, and the transferability of synthetic deliberation trajectories. None of these are proven by the paper; they are standard assumptions for this class of agent-tuning method. The only tuned coefficients are N, I, temperature, and SFT hyperparameters, all chosen by hand.

free parameters (4)
  • sample count N = N=5 for ALFWorld/ScienceWorld; N=3 for WebShop
    Hand-chosen number of sampled candidate actions per deliberation step; no sensitivity analysis is reported.
  • iteration count I = I=3
    Stopping point for iterative self-training; results are non-monotonic in some table cells, suggesting saturation or mild instability.
  • action sampling temperature = 1.0
    Used for self-consistency action sampling and rollout execution; critique and synthesis use temperature 0, and the value is not swept.
  • SFT hyperparameters = batch size 64, learning rate 1e-5, 3 epochs for iteration 1 and 1 epoch for later iterations
    Training choices inherited from the ETO setup; their effect on SAND is not analyzed.
assumptions (4)
  • domain assumption Expert ReAct trajectories are high-quality and representative of optimal behavior.
    SFT initialization and deliberation synthesis both treat expert actions as ground truth, as stated in Sections 4.1 and 4.4.
  • domain assumption Environment reward r(u,e) in [0,1] is a reliable signal for step-level action critique.
    Execution-guided critique uses final task reward from rollouts to judge each candidate action, despite rewards being delayed and sparse in ALFWorld and ScienceWorld.
  • domain assumption N samples from the current policy at temperature 1.0 adequately cover the decision-relevant action space around expert trajectories.
    Self-consistency sampling assumes the set of sampled actions plus the expert action reveals genuine uncertainty; no coverage guarantee is given for large action spaces.
  • domain assumption Supervised finetuning on self-synthesized deliberation trajectories transfers to unseen test tasks without reward overfitting.
    The whole method relies on generalization from synthetic trajectories; only test-set rewards are reported, with no distribution-shift analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAND: Boosting LLM Agents with Self-Taught Action Deliberation." pith.science (2026). https://pith.science/paper/YOCFBCT4

@misc{pith2026250707441,
  author       = {Pith},
  title        = {Pith review of: SAND: Boosting LLM Agents with Self-Taught Action Deliberation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOCFBCT4}},
  note         = {Machine review of arXiv:2507.07441}
}
read the original abstract

Large Language Model (LLM) agents are commonly tuned with supervised finetuning on ReAct-style expert trajectories or preference optimization over pairwise rollouts. Most of these methods focus on imitating specific expert behaviors or promoting chosen reasoning thoughts and actions over rejected ones. However, without reasoning and comparing over alternatives actions, LLM agents finetuned with these methods may over-commit towards seemingly plausible but suboptimal actions due to limited action space exploration. To address this, in this paper we propose Self-taught ActioN Deliberation (SAND) framework, enabling LLM agents to explicitly deliberate over candidate actions before committing to one. To tackle the challenges of when and what to deliberate given large action space and step-level action evaluation, we incorporate self-consistency action sampling and execution-guided action critique to help synthesize step-wise action deliberation thoughts using the base model of the LLM agent. In an iterative manner, the deliberation trajectories are then used to finetune the LLM agent itself. Evaluating on two representative interactive agent tasks, SAND achieves an average 20% improvement over initial supervised finetuning and also outperforms state-of-the-art agent tuning approaches.

Figures

Figures reproduced from arXiv: 2507.07441 by the authors.

Figure 1
Figure 1. An illustrative example of an LLM agent task, where SFT trained agent (Zeng et al., 2024) over￾commits to an seemingly plausible but suboptimal ac￾tion while our SAND tuned agent learns to deliberate over candidate actions before choosing the best action. information in context. To obtain more reliable LLM agents, recent works apply supervised fine￾tuning on expert ReAct-style trajectories (Chen et al., 2023; Zeng e… view at source ↗
Figure 2
Figure 2. An illustration of our SAND framework for synthesizing one step of action deliberation thoughts. 2.3 Iterative Self Learning Another relevant line of works enable a model to improve by repeatedly generating data and fine￾tuning on its own synthesized output (Xia et al., 2025b). The idea began with STaR (Zelikman et al., 2022), which bootstraps a few verified solutions into a large corpus of correct rationales. RFT (… view at source ↗
Figure 3
Figure 3. Average reward per step (bars) and average action deliberation rate per step (lines) on test sets. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Action deliberation rate distribution across three difficulty bands in unseen test set on ScienceWorld. Each [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt used for the execution-guided action critique. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt used for action deliberation synthesis. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt used for ALFWorld tasks [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Prompt used for ScienceWorld tasks [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

    arXiv 2023

  5. [5]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 9354--9366

    work page 2024

  6. [6]

    Zhixun Chen, Ming Li, Yuxuan Huang, Yali Du, Meng Fang, and Tianyi Zhou. 2025. Atlas: Agent tuning via learning critical steps. arXiv preprint arXiv:2503.02197

    arXiv 2025

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339

    arXiv 2024

Show all 39 references
  1. [9]

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143

    2024 arXiv

  2. [10]

    Arjun Karanam, Farnaz Jahanbakhsh, and Sanmi Koyejo. 2024. Towards deliberating agents: Evaluating the ability of large language models to deliberate. In NeurIPS 2024 Workshop on Behavioral Machine Learning

    2024

  3. [11]

    Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. 2024. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407.14507

    2024 arXiv

  4. [12]

    Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. 2025. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584

    2025 arXiv

  5. [13]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534--46594

    2023

  6. [14]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  7. [15]

    Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2024. Agent planning with world knowledge model. Advances in Neural Information Processing Systems, 37:114843--114871

    2024

  8. [16]

    Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.138 Direct multi-turn preference optimization for language agents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page...

    2024 doi

  9. [17]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634--8652

    2023

  10. [18]

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C \^o t \'e , Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768

    2020 arXiv

  11. [19]

    Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024 a . Agentbank: Towards generalized llm agents via fine-tuning on 50000+ interaction trajectories. In Findings of the Association for Computational Linguistics: EMNLP ...

    2024

  12. [20]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.409 Trial and error: Exploration-based trajectory optimization of LLM agents . In Proceedings of the 62nd Annual Meeting of the Association for Computat...

    2024 doi

  13. [21]

    Renxi Wang, Xudong Han, Yixuan Zhang, Timothy Baldwin, and Haonan Li. 2025. https://aclanthology.org/2025.naacl-long.378/ NAT : Enhancing agent tuning with negative samples . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Co...

    2025

  14. [22]

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540

    2022 arXiv

  15. [23]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferenc...

    2023

  16. [24]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  17. [25]

    Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2025. Agentrm: Enhancing agent generalization with reward modeling. arXiv preprint arXiv:2502.18407

    2025 arXiv

  18. [26]

    Siheng Xiong, Ali Payani, Yuan Yang, and Faramarz Fekri. 2024 a . Deliberate reasoning for llms as structure-aware planning with accurate world model. arXiv preprint arXiv:2410.03136

    2024 arXiv

  19. [27]

    Weimin Xiong, Yifan Song, Qingxiu Dong, Bingchan Zhao, Feifan Song, Xun Wang, and Sujian Li. 2025. Mpo: Boosting llm agents with meta plan optimization. arXiv preprint arXiv:2503.02682

    2025 arXiv

  20. [28]

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.93 Watch every step! LLM agent learning via iterative step-level process refinement . In Proceedings of the 2024 Confer...

    2024 doi

  21. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  22. [30]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744--20757

    2022

  23. [31]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023 a . Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  24. [32]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023 b . https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  25. [33]

    Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. 2025. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425

    2025 arXiv

  26. [34]

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825

    2023 arXiv

  27. [35]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

    2022

  28. [36]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.181 A gent T uning: Enabling generalized agent abilities for LLM s . In Findings of the Association for Computational Linguistics: ACL 2024, pa...

    2024 doi

  29. [37]

    Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang, Bo Ding, and Huaimin Wang. 2025. Enhancing decision-making for llm agents via step-level q-value models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27161--27169

    2025

  30. [38]

    Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. A survey on multi-turn interaction capabilities of large language models. arXiv preprint arXiv:2501.09959

    2025 arXiv

  31. [39]

    Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. 2025. https://aclanthology.org/2025.findings-naacl.205/ K now A gent: Knowledge-augmented planning for LLM -based agents . In Findings of the Association f...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.