{"id":"24292877-4f1a-4985-870a-3e11740bf6e8","arxiv_id":"2411.19547","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Iteratively fine-tuning LLM agents on trajectories selected by a critic LLM yields API-Bank accuracy close to GPT-4 using much smaller models.","lead":"A team at Tencent trains small open-source language models to act as API-calling agents by repeatedly sampling their own attempts, having a larger LLM pick the best ones, and fine-tuning on those picks. On the API-Bank benchmark their 6B and 13B models reach roughly 48 to 50 percent accuracy, close to GPT-4's 51.6 percent, without using expert-provided trajectories.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The missing random-trajectory SFT baseline means the critic's selection, rather than mere SFT on self-generated API trajectories, is not established as the driver of the reported gains.","rationale":"The reader's weakest assumption is exactly the missing link between critic scores and downstream task success: the paper does not test whether training on critic-selected trajectories outperforms random-trajectory SFT. My stress-test agrees that this is the most load-bearing concern. If random-trajectory SFT matches the reported accuracy, the entire 'weakly supervised feedback' mechanism collapses, and the result reduces to ordinary fine-tuning on generated trajectories. If random SFT is clearly worse, the critic selection is validated as the key component. The concern is concrete and testable, and it is not a disagreement with the field's consensus but a missing control in the paper's own argument. The small 95-question test set and absent error bars amplify the risk: the gap between Ours-Yi-6B (47.5%) and GPT-4 (51.6%) is only about four questions, so without confidence intervals even the headline comparison is fragile. The Equation (1) inconsistency also weakens confidence that the implemented training objective matches the described top-p% selection. Because the reader already marked the paper CONDITIONAL and this concern is the same one, no verdict change is needed; the paper should remain CONDITIONAL pending the random-selection baseline and ideally error bars or a larger evaluation set.","tokens_in":6572,"tokens_out":4781,"duration_ms":45617,"concrete_test":"Run a control experiment: with the same 220 API-Bank training instructions and the same Yi-6B actor, generate K=5 trajectories per instruction, then randomly select the same number of trajectories per round as the top-10% condition (also excluding previously used trajectories), fine-tune with identical hyperparameters (LR 5e-5 cosine decay to 5e-6, 1:1 general chat data mix) for the same number of iterations, and evaluate on the same 95 held-out questions. Report accuracy with bootstrap 95% confidence intervals. If random-selection SFT reaches roughly 47.5% (within the confidence interval), the critic selection is not the driver; if it is substantially lower, the weak-supervision claim is supported. Additionally, inspect the released training code to confirm that Equation (1) is applied only to the selected top-p% trajectories and not to all generated trajectories.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that iterative fine-tuning on critic-selected top-10% trajectories lets agents improve under weak supervision. The load-bearing condition is that the critic's selection, not SFT on any self-generated trajectory set, drives the improvement from 10.5% (Yi-6B) to 47.5% (Ours-Yi-6B). This is not established: there is no control experiment using randomly selected trajectories of the same count, with identical SFT hyperparameters and iteration count. The critic confusion matrix (Table 1, Section 3.2) reports 70% precision on a random sample and does not measure precision on the top-10% high-confidence set actually used for training, so the quality of the training filter is also uncharacterized. Additionally, Section 2.4 says only the top p% of trajectories are selected, but the training loss in Equation (1) sums over all N×K trajectories; if taken literally, the objective does not implement the described filtering, making the mechanism ambiguous. A random-selection baseline would directly test whether the critic is load-bearing or whether the gains come from fine-tuning on any distribution of API-Bank-like trajectories.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an iterative training framework for LLM-based agents under weak supervision. The actor samples K trajectories per instruction through interaction with a tool environment; a critic LLM (Yi-34B) scores the trajectories; the top p% are selected and used for supervised fine-tuning together with general chat data; the process is repeated over several iterations. The method is evaluated on the API-Bank benchmark, reporting 47.5% accuracy for Ours-Yi-6B and 49.5% for Ours-Llama2-13B, close to GPT-4's 51.6%, with base models at 10.5% and 15.8% respectively. The paper claims that this is achieved without expert trajectories or definitive environmental feedback, and that iterative selection of high-confidence trajectories mitigates critic error.","tokens_in":6834,"tokens_out":3450,"duration_ms":32169,"significance":"If the reported gains are reproducible, this is a useful contribution: it shows that a smaller open-source model can approach a much larger commercial model on a tool-use benchmark using only weak, LLM-generated feedback, which could reduce the need for expensive expert demonstrations. The evaluation on a public benchmark with human-verified held-out accuracy is a positive feature, as is the explicit attempt to report critic-human agreement in a confusion matrix. However, the current empirical support is thin: there are no confidence intervals or multiple seeds, only 95 test questions, no random-trajectory baseline, and the critic's precision on the actual training filter is not measured. The central claim is defensible but not yet established by the evidence in the manuscript.","major_comments":[{"comment":"The paper's central claim is that critic-based selection of top-10% trajectories drives the large improvement from 10.5% (Yi-6B) to 47.5% (Ours-Yi-6B). This is not established because there is no control experiment that fine-tunes the actor on randomly selected self-generated trajectories of the same count, using the same SFT hyperparameters and the same number of iterations. Without such a baseline, the gains could be attributed to supervised fine-tuning on API-Bank-like trajectories generally, rather than to the critic's filtering. I request a random-selection SFT ablation (and ideally also a no-filter SFT on all sampled trajectories) to make the role of the critic load-bearing.","section":"Section 3.3, Table 2; Section 2.4"},{"comment":"The training objective in Eq. (1) sums over all N×K trajectories with equal weight, but the text states that only the top p% of trajectories are selected for training and that previously used trajectories are excluded. If Eq. (1) is taken literally, it does not implement the described selection mechanism. The equation should be written over the selected subset, or the notation should make explicit which indicator function or subset is used; as written, the description of the training signal is internally inconsistent.","section":"Section 2.4, Eq. (1)"},{"comment":"The critic evaluation reports 70% precision on a randomly selected sample of trajectories, but the training procedure uses only the top-10% highest-scoring trajectories. Precision on a random sample is not the relevant quantity: the filter quality for training depends on precision on the high-confidence top-10% set. The paper should report how often trajectories in this selected set are actually correct (or a human-verified comparison between selected and unselected trajectories). The current numbers leave open the possibility that the critic's selection has low precision exactly on the subset used for training.","section":"Section 3.2, Table 1"},{"comment":"All accuracy results are reported without confidence intervals, variance across seeds, or statistical significance tests. With only 95 held-out questions, each question is approximately 1.05 percentage points, and the difference between Ours-Yi-6B (47.5%) and Ours-Llama2-13B (49.5%) is about two questions. The claim of 'comparable performance to GPT-4' and the comparison across training iterations would be substantially stronger with multiple independent runs or bootstrap confidence intervals. I recommend adding at least two or three seeds for the main results and pairwise significance information.","section":"Section 3.3, Figure 2, Table 2"}],"minor_comments":[{"comment":"The paper repeatedly refers to detailed prompts and trajectory examples in the Appendix, but the Appendix is empty. Please include the critic prompt, the actor/reasoning prompt, the ReAct prompt, at least one full trajectory example, and the exact instruction set used for trajectory sampling.","section":"Appendix"},{"comment":"The '1:1 ratio between trajectory data and the general data' is ambiguous: it is not clear whether the ratio applies after top-p% selection, after excluding previously used trajectories, or to the entire pool of generated trajectories. Please specify the exact data composition used for each SFT step.","section":"Section 2.4"},{"comment":"The sentence 'we evaluate the accuracy of this module by randomly selecting a number of samples rated by the critic' does not report how many samples were selected. Please provide the sample size used for the confusion matrix in Table 1.","section":"Section 3.2"},{"comment":"Some references are incomplete, for example 'Kadlčík et al.;' appears without a year or venue, and the Llama 2 citation appears twice with different reference keys (Touvron et al., 2023a and 2023b). Please clean up the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable candidate for a workshop or conference focused on LLM agents, but the reproducibility issues are substantial: no code, no prompts, no seeds, and an empty appendix. The missing random-selection baseline is the most important technical gap, and I would not recommend acceptance until it is addressed. The relation to prior self-improvement and weak-supervision methods (e.g., STaR, Self-Refine, CRITIC, LLaMA-Rider) is also underexplored and should be discussed honestly; the novelty of the method rests mainly on the specific iterative critic-selection design, which is a modest but useful step."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a plausible self-training recipe for tool-use agents, and the headline number on API-Bank is worth taking seriously, but the experiments as presented leave the mechanism underdetermined. The actual contribution is narrow: iterative SFT on critic-LLM-selected trajectories, with Yi-34B as critic and Yi-6B/Llama2-13B as actors, reaching 47.5% and 49.5% vs GPT-4's 51.6%. The core loop is straight from the self-training literature (STaR, Huang et al., etc.), so novelty is modest, but the specific combination on API-Bank is new and the gain from 10.5% to 47.5% is large enough to merit attention.\n\nWhat the paper does well: the evaluator is human verification on held-out questions, so the headline is not circular. The limitations section openly states that the critic's precision is limited. Figure 2 shows monotone improvement across epochs, consistent with the story.\n\nWhere it is soft, in proportion: the load-bearing claim is that the critic's selection, not just SFT on any self-generated trajectories, drives the improvement. There is no random-trajectory SFT control, so the critic's role is not isolated. The stress-test note is correct on this point. Related, the critic's 70% precision is measured on a random sample; precision on the top-10% high-confidence set actually used for training is not reported, and that is the number that matters. Also, Section 2.4 says only top-p% of trajectories are selected, but Equation (1) sums over all N×K trajectories; as written the loss does not implement the filtering. Probably sloppy notation, but it needs fixing. The test set is 95 questions, with no confidence intervals, seeds, or code release, and the appendix with prompts is empty, so independent reruns are not currently possible. These are fixable but real gaps.\n\nFor whom: people working on self-improving agents and weak-supervision training will get value from it. It deserves a serious referee; the question is important and the result, if confirmed, is useful. I would not cite it in its current form, but I would engage with it and ask for the missing baselines, error bars, and the actual prompts. The core idea is sound.","headline":"A plausible self-training recipe for tool agents with a headline gain that the current experiments do not fully pin down.","tokens_in":7364,"tokens_out":3070,"would_cite":false,"duration_ms":24925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A critic LLM's feedback alone can replace expert demonstrations for training tool-using agents, nearly matching GPT-4.","keywords":["LLM agents","weakly supervised feedback","critic LLM","iterative self-improvement","trajectory selection","supervised fine-tuning","tool use","API-Bank"],"falsifier":"Run the same evolution loop on API-Bank but replace the top-10% selection with a random 10% of scored trajectories in each round. If the random-control model reaches the same roughly 47-49% accuracy, the paper's explanation of the gains—critic-selected high-quality training data—is false.","tokens_in":6395,"feed_emoji":"🤖","tokens_out":6247,"duration_ms":52856,"temperature":0.7,"pith_summary":"This paper argues that an LLM-based agent can be trained to call APIs and follow instructions without expert-written trajectories or a hard environmental reward. The method runs an iterative loop: the agent samples several trajectories, a critic LLM scores them, and only the top 10% are used for supervised fine-tuning, repeated over iterations. On the API-Bank benchmark, the paper reports steady accuracy gains, reaching 47.5% for a 6B model and 49.5% for a 13B model, within a few points of GPT-4's 51.6%. The point matters because weak, qualitative feedback from an LLM judge is available in many domains where definitive reward signals are not.","feed_headline":"Critic feedback alone lifts small agent models close to GPT-4","feed_subtitle":"Iterative fine-tuning on the top 10% of critic-scored trajectories lifts a 6B model from 10.5% to 47.5% on API-Bank.","key_machinery":"The load-bearing mechanism is a three-module evolution loop: an actor LLM samples $K=5$ trajectories per instruction against an API environment, a critic LLM (Yi-34B) scores each trajectory with a hand-designed prompt, and a trainer performs supervised fine-tuning on the top 10% of scored trajectories while excluding previously used ones and adding general chat data at a 1:1 ratio. The training objective is the standard negative log-likelihood of the selected action sequence, written as $\\mathcal{L} = -\\log P_\\theta(A_m \\mid I, A_1, O_1, \\dots, A_{m-1}, O_{m-1})$ and summed over selected trajectories and steps. The iterative re-sampling makes the selection high-confidence and lets the actor explore and gradually absorb critic feedback.","core_discovery":"The central claim is that iterative supervised fine-tuning on a small, high-scoring subset of an agent's own interaction trajectories is sufficient for agent improvement, provided a capable LLM critic supplies the scores. The authors show this on API-Bank using Yi-6B and Llama2-13B as actors and Yi-34B as critic: across five evolution rounds, accuracy on 95 held-out questions rises from 10.5% to 47.5% for Yi-6B and from 15.8% to 49.5% for Llama2-13B, compared with GPT-4's 51.6%. They also report that the critic agrees with human evaluation on 70% of flagged successes while capturing 97.2% of true successes, and they argue that the iterative recipe—sampling five trajectories per instruction, keeping the top 10% each round, and mixing in general chat data—compensates for critic noise. The setup deliberately avoids both imitation of expert demonstrations and reinforcement learning from definitive rewards.","pith_inferences":["Editorial inference: the paper does not run a random-trajectory control, so part of the observed gain could come from fine-tuning on any interaction trajectories rather than from the critic's selection.","Editorial inference: the loop's ceiling likely depends on the critic being stronger than the actor; a critic no stronger than the actor may stall, and monitoring per-iteration critic precision could reveal when.","Editorial inference: the same recipe could be tested in other weakly supervised settings, such as web navigation or database querying, where an LLM judge can rank trajectories but no reward exists."],"forward_implications":["If the claim holds, agent training in tool-use environments no longer requires expert demonstrations or handcrafted reward functions.","The same loop should transfer to other domains where a judge LLM can rank multi-step behavior, such as customer service, database queries, or web navigation.","Smaller open-weight models can approach the performance of larger proprietary models on a benchmark solely through self-generated data.","Because only high-confidence trajectories are used, imperfect critic precision still permits improvement across iterations."],"supporting_citations":[{"why":"Supplies the API-Bank benchmark data used for evolution training and the API-Bank-7B baseline checkpoint compared in the experiments.","marker":"(Li et al., 2023)"},{"why":"Provides the Yi model family, which supplies both the Yi-6B actor and the Yi-34B critic, as well as the Yi-6B and Yi-34B baseline results.","marker":"(Young et al., 2024)"},{"why":"Provides the Llama2-13B base model used for the second actor and its reported baseline accuracy.","marker":"(Touvron et al., 2023b)"},{"why":"Supplies the ReAct prompting method and its baseline accuracy, which the trained agent outperforms in the benchmark comparison.","marker":"(Yao et al., 2022)"}],"fun_headline_variants":["Critic feedback trains agents without expert demos or rewards","Iterative critic scoring lifts small agents near GPT-4 performance","Weakly supervised LLM critic improves agents iteratively","No expert demos needed: critic picks best trajectories for training","Small agent models approach GPT-4 via critic-scored evolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the critic LLM's top-10% scores pick out trajectories that are genuinely more worth learning from; if the top 10% were no better than a random tenth, the reported gains would have to be explained by something else.","fun_headline_variants_meta":{"raw":{"variants":["Critic feedback trains agents without expert demos or rewards","Iterative critic scoring lifts small agents near GPT-4 performance","Weakly supervised LLM critic improves agents iteratively","No expert demos needed: critic picks best trajectories for training","Small agent models approach GPT-4 via critic-scored evolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1789,"prompt_tokens":916,"completion_tokens":873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":790}},"tokens_in":532,"tokens_out":873,"duration_ms":7254,"temperature":1.0,"reasoning_tokens":790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:04:48.650175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same evolution loop on API-Bank but replace the top-10% selection with a random 10% of scored trajectories in each round. If the random-control model reaches the same roughly 47-49% accuracy, the paper's explanation of the gains—critic-selected high-quality training data—is false.","supporting_citations":[],"review_version":1}