Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Training Agents with Weakly Supervised Feedback from Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A critic LLM's feedback alone can replace expert demonstrations for training tool-using agents, nearly matching GPT-4.

desk verdict A plausible self-training recipe for tool agents with a headline gain that the current experiments do not fully pin down. read the letter →

arxiv 2411.19547 v1 pith:R5ER4KTK submitted 2024-11-29 cs.CL cs.AI

Dihong Gong , Pu Lu , Zelong Wang , Meng Zhou , Xiuqiang He This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMagentsweaklysupervisedfeedbackcriticiterativeself-improvementtrajectoryselectionfine-tuningtooluseAPI-Bank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an LLM-based agent can be trained to call APIs and follow instructions without expert-written trajectories or a hard environmental reward. The method runs an iterative loop: the agent samples several trajectories, a critic LLM scores them, and only the top 10% are used for supervised fine-tuning, repeated over iterations. On the API-Bank benchmark, the paper reports steady accuracy gains, reaching 47.5% for a 6B model and 49.5% for a 13B model, within a few points of GPT-4's 51.6%. The point matters because weak, qualitative feedback from an LLM judge is available in many domains where definitive reward signals are not.

What carries the argument

The load-bearing mechanism is a three-module evolution loop: an actor LLM samples $K=5$ trajectories per instruction against an API environment, a critic LLM (Yi-34B) scores each trajectory with a hand-designed prompt, and a trainer performs supervised fine-tuning on the top 10% of scored trajectories while excluding previously used ones and adding general chat data at a 1:1 ratio. The training objective is the standard negative log-likelihood of the selected action sequence, written as $\mathcal{L} = -\log P_\theta(A_m \mid I, A_1, O_1, \dots, A_{m-1}, O_{m-1})$ and summed over selected trajectories and steps. The iterative re-sampling makes the selection high-confidence and lets the actor explore and gradually absorb critic feedback.

What would settle it

Run the same evolution loop on API-Bank but replace the top-10% selection with a random 10% of scored trajectories in each round. If the random-control model reaches the same roughly 47-49% accuracy, the paper's explanation of the gains—critic-selected high-quality training data—is false.

Watch

Extended reading notes

Core claim

The central claim is that iterative supervised fine-tuning on a small, high-scoring subset of an agent's own interaction trajectories is sufficient for agent improvement, provided a capable LLM critic supplies the scores. The authors show this on API-Bank using Yi-6B and Llama2-13B as actors and Yi-34B as critic: across five evolution rounds, accuracy on 95 held-out questions rises from 10.5% to 47.5% for Yi-6B and from 15.8% to 49.5% for Llama2-13B, compared with GPT-4's 51.6%. They also report that the critic agrees with human evaluation on 70% of flagged successes while capturing 97.2% of true successes, and they argue that the iterative recipe—sampling five trajectories per instruction, keeping the top 10% each round, and mixing in general chat data—compensates for critic noise. The setup deliberately avoids both imitation of expert demonstrations and reinforcement learning from definitive rewards.

Load-bearing premise

The load-bearing premise is that the critic LLM's top-10% scores pick out trajectories that are genuinely more worth learning from; if the top 10% were no better than a random tenth, the reported gains would have to be explained by something else.

Editorial extensions

If this is right

  • If the claim holds, agent training in tool-use environments no longer requires expert demonstrations or handcrafted reward functions.
  • The same loop should transfer to other domains where a judge LLM can rank multi-step behavior, such as customer service, database queries, or web navigation.
  • Smaller open-weight models can approach the performance of larger proprietary models on a benchmark solely through self-generated data.
  • Because only high-confidence trajectories are used, imperfect critic precision still permits improvement across iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper does not run a random-trajectory control, so part of the observed gain could come from fine-tuning on any interaction trajectories rather than from the critic's selection.
  • Editorial inference: the loop's ceiling likely depends on the critic being stronger than the actor; a critic no stronger than the actor may stall, and monitoring per-iteration critic precision could reveal when.
  • Editorial inference: the same recipe could be tested in other weakly supervised settings, such as web navigation or database querying, where an LLM judge can rank trajectories but no reward exists.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes an iterative training framework for LLM-based agents under weak supervision. The actor samples K trajectories per instruction through interaction with a tool environment; a critic LLM (Yi-34B) scores the trajectories; the top p% are selected and used for supervised fine-tuning together with general chat data; the process is repeated over several iterations. The method is evaluated on the API-Bank benchmark, reporting 47.5% accuracy for Ours-Yi-6B and 49.5% for Ours-Llama2-13B, close to GPT-4's 51.6%, with base models at 10.5% and 15.8% respectively. The paper claims that this is achieved without expert trajectories or definitive environmental feedback, and that iterative selection of high-confidence trajectories mitigates critic error.

Significance. If the reported gains are reproducible, this is a useful contribution: it shows that a smaller open-source model can approach a much larger commercial model on a tool-use benchmark using only weak, LLM-generated feedback, which could reduce the need for expensive expert demonstrations. The evaluation on a public benchmark with human-verified held-out accuracy is a positive feature, as is the explicit attempt to report critic-human agreement in a confusion matrix. However, the current empirical support is thin: there are no confidence intervals or multiple seeds, only 95 test questions, no random-trajectory baseline, and the critic's precision on the actual training filter is not measured. The central claim is defensible but not yet established by the evidence in the manuscript.

major comments (4)
  1. [Section 3.3, Table 2; Section 2.4] The paper's central claim is that critic-based selection of top-10% trajectories drives the large improvement from 10.5% (Yi-6B) to 47.5% (Ours-Yi-6B). This is not established because there is no control experiment that fine-tunes the actor on randomly selected self-generated trajectories of the same count, using the same SFT hyperparameters and the same number of iterations. Without such a baseline, the gains could be attributed to supervised fine-tuning on API-Bank-like trajectories generally, rather than to the critic's filtering. I request a random-selection SFT ablation (and ideally also a no-filter SFT on all sampled trajectories) to make the role of the critic load-bearing.
  2. [Section 2.4, Eq. (1)] The training objective in Eq. (1) sums over all N×K trajectories with equal weight, but the text states that only the top p% of trajectories are selected for training and that previously used trajectories are excluded. If Eq. (1) is taken literally, it does not implement the described selection mechanism. The equation should be written over the selected subset, or the notation should make explicit which indicator function or subset is used; as written, the description of the training signal is internally inconsistent.
  3. [Section 3.2, Table 1] The critic evaluation reports 70% precision on a randomly selected sample of trajectories, but the training procedure uses only the top-10% highest-scoring trajectories. Precision on a random sample is not the relevant quantity: the filter quality for training depends on precision on the high-confidence top-10% set. The paper should report how often trajectories in this selected set are actually correct (or a human-verified comparison between selected and unselected trajectories). The current numbers leave open the possibility that the critic's selection has low precision exactly on the subset used for training.
  4. [Section 3.3, Figure 2, Table 2] All accuracy results are reported without confidence intervals, variance across seeds, or statistical significance tests. With only 95 held-out questions, each question is approximately 1.05 percentage points, and the difference between Ours-Yi-6B (47.5%) and Ours-Llama2-13B (49.5%) is about two questions. The claim of 'comparable performance to GPT-4' and the comparison across training iterations would be substantially stronger with multiple independent runs or bootstrap confidence intervals. I recommend adding at least two or three seeds for the main results and pairwise significance information.
minor comments (4)
  1. [Appendix] The paper repeatedly refers to detailed prompts and trajectory examples in the Appendix, but the Appendix is empty. Please include the critic prompt, the actor/reasoning prompt, the ReAct prompt, at least one full trajectory example, and the exact instruction set used for trajectory sampling.
  2. [Section 2.4] The '1:1 ratio between trajectory data and the general data' is ambiguous: it is not clear whether the ratio applies after top-p% selection, after excluding previously used trajectories, or to the entire pool of generated trajectories. Please specify the exact data composition used for each SFT step.
  3. [Section 3.2] The sentence 'we evaluate the accuracy of this module by randomly selecting a number of samples rated by the critic' does not report how many samples were selected. Please provide the sample size used for the confusion matrix in Table 1.
  4. [References] Some references are incomplete, for example 'Kadlčík et al.;' appears without a year or venue, and the Llama 2 citation appears twice with different reference keys (Touvron et al., 2023a and 2023b). Please clean up the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the held-out benchmark accuracy is human-verified and independent of the critic LLM's training signal.

full rationale

The paper's central claim is that iterative fine-tuning on critic-selected trajectories improves agent accuracy on API-Bank. The reported evaluation is held-out accuracy on 95 questions whose answers are verified by human evaluators (Section 3.1: 'The accuracy is determined by the correctness of the answers, which are reviewed and verified by human evaluators'), not by the critic module used to select training trajectories. The critic (Yi-34B) only ranks trajectories for SFT; it is not fitted to the test set, and its scores are not the measured outcome. There is no fitted parameter renamed as a prediction, no load-bearing self-citation chain, and no uniqueness theorem imported from the authors' prior work. The apparent mismatch between the top-10% selection described in Section 2.4 and the training loss in Equation (1), which sums over all N×K trajectories, is an internal inconsistency or reporting ambiguity, but it does not make the reported improvements equal to the method's inputs by construction. The limitation section explicitly acknowledges that critic precision is limited, which further indicates the benchmark outcome is not guaranteed by the critic's preferences. Therefore, no circular step is identifiable under the strict reduction criterion.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set hyperparameters (K, M, top-10%, data ratio, iterations, learning rate) and on the unverified assumption that critic-filtered trajectories are better training data than random trajectories. No new theoretical entities are introduced.

free parameters (6)
  • K (trajectories sampled per instruction) = 5
    Chosen in Section 2.5; determines exploration diversity and cost, and influences how often good trajectories appear for the critic to select.
  • M (maximum interaction rounds) = 5
    Chosen in Section 2.5; caps trajectory length and affects which instructions can be completed.
  • top-p% of trajectories selected by critic = 10%
    Chosen in Section 2.5; controls the tradeoff between training signal size and label noise. The paper does not ablate this value.
  • chat data ratio = 1:1
    Chosen in Section 2.4 to prevent overfitting; not ablated.
  • number of evolution iterations = 4 (epochs 0-4 in Figure 2)
    Stopping point chosen after observing Figure 2; no criterion for convergence is given.
  • learning rate schedule = 5e-5 to 5e-6 cosine, no warm-up
    Chosen in Section 2.5; standard for SFT, but still a hand-set hyperparameter.
assumptions (4)
  • domain assumption The critic LLM's trajectory scores are a meaningful signal of trajectory quality under the human-verified accuracy metric.
    Section 2.3 and Table 1; precision is only 70%, so the method assumes the top-10% high-score trajectories are beneficial enough for SFT despite the noise.
  • domain assumption The current agent's sampled trajectories are informative enough to bootstrap improvement, even when initial accuracy is low.
    Section 2.2 and Figure 2; if sampling rarely produced good trajectories, the top-10% selection would be empty or too noisy.
  • domain assumption Standard language-model negative log-likelihood fine-tuning on selected trajectories transfers to improved interaction accuracy.
    Section 2.4, Eq. (1); the paper provides no mechanism beyond SFT to explain why the trained actor should generalize on held-out questions.
  • domain assumption The 95-question held-out split and the human-reviewed accuracy metric reliably measure agent capability.
    Section 3.1; no confidence intervals, no inter-annotator agreement, and no repeated-seed variance are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Agents with Weakly Supervised Feedback from Large Language Models." pith.science (2026). https://pith.science/paper/R5ER4KTK

@misc{pith2026241119547,
  author       = {Pith},
  title        = {Pith review of: Training Agents with Weakly Supervised Feedback from Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5ER4KTK}},
  note         = {Machine review of arXiv:2411.19547}
}
read the original abstract

Large Language Models (LLMs) offer a promising basis for creating agents that can tackle complex tasks through iterative environmental interaction. Existing methods either require these agents to mimic expert-provided trajectories or rely on definitive environmental feedback for reinforcement learning which limits their application to specific scenarios like gaming or code generation. This paper introduces a novel training method for LLM-based agents using weakly supervised signals from a critic LLM, bypassing the need for expert trajectories or definitive feedback. Our agents are trained in iterative manner, where they initially generate trajectories through environmental interaction. Subsequently, a critic LLM selects a subset of good trajectories, which are then used to update the agents, enabling them to generate improved trajectories in the next iteration. Extensive tests on the API-bank dataset show consistent improvement in our agents' capabilities and comparable performance to GPT-4, despite using open-source models with much fewer parameters.

Figures

Figures reproduced from arXiv: 2411.19547 by the authors.

Figure 1
Figure 1. Our self-evolving algorithm employs a comprehensive training pipeline to instruct LLMs in the utilization [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy results under different number of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 5 canonical work pages

  1. [1]

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys aw D e biak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680

    arXiv 2019

  2. [2]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

    arXiv 2023

  3. [3]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588

    arXiv 2022

  4. [4]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024. https://api.semanticscholar.org/CorpusID:268532485 Agent-flan: Designing data and methods of effective agent tuning for large language models . ArXiv, abs/2403.12881

    arXiv 2024

  5. [5]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Tora: A tool-integrated reasoning agent for mathematical problem solving

    work page 2023

  6. [6]

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856

    arXiv 2023

  7. [7]

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.11610

    arXiv 2022

  8. [8]

    Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Selfevolve: A code evolution framework via large language models. arXiv preprint arXiv:2306.02907

    arXiv 2023

Show all 33 references
  1. [9]

    Self-training language models in arithmetic reasoning

    Marek Kadl c \' k, Michal S tef \'a nik, Ondrej Sotolar, and Vlastimil Martinek. Self-training language models in arithmetic reasoning. In ICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  2. [10]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  3. [11]

    Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. 2024. Mario: Math reasoning with code interpreter output--a reproducible pipeline. arXiv preprint arXiv:2401.08190

    2024 arXiv

  4. [12]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations

    2023

  5. [13]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  6. [14]

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pages 26106--26128. PMLR

    2023

  7. [15]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334

    2023 arXiv

  8. [16]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36

    2024

  9. [17]

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604--609

    2020

  10. [18]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  11. [19]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press

    2018

  13. [21]

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301

    2023 arXiv

  14. [22]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  15. [23]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  16. [24]

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha \"e l Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350--354

    2019

  17. [25]

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935

    2023 arXiv

  18. [26]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864

    2023 arXiv

  19. [27]

    Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. 2023. Lemur: Harmonizing natural language and code for language agents. arXiv preprint arXiv:2310.06830

    2023 arXiv

  20. [28]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629

    2022 arXiv

  21. [29]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  22. [30]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823

    2023 arXiv

  23. [31]

    Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. 2024. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint arXiv:2402.17574

    2024 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.