{"id":"0109ed95-7f5d-42ac-8f71-1e290c1cd553","arxiv_id":"2507.14447","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured planning format called Routine sharply improves LLM agents' multi-step tool-calling accuracy in an enterprise scenario.","lead":"Routine is a structured, step-by-step planning format that tells an AI agent which tools to call and in what order. In an enterprise HR test, it raised GPT-4o's tool-calling accuracy from 41.1% to 96.3%, and a smaller Qwen3-14B model from 32.6% to 83.3%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains (41.1%→96.3%) partly reflect a task-reformulation artifact: the Routine condition supplies the correct tool name for every step, so the comparison conflates 'better planning' with 'removing the need to plan'.","rationale":"The reader correctly identifies the self-distillation and oracle-history issues, and those remain serious threats to the quantitative headline. However, the single most load-bearing concern for the paper's central claim—that the Routine structure improves multi-step tool calling—is that the comparison is confounded: the Routine condition gives away the correct tool name at every step. The ablation in §5.3.1 demonstrates that the tool name is the most important component, so the headline 41.1%→96.3% largely reflects replacing autonomous tool selection with literal instruction-following. This is not a fatal flaw: the paper's contribution can honestly be framed as a prompt/workflow engineering recipe, and the fine-tuning results (scenario-trained Qwen3-14B reaching 95.5%) are less affected by this confound because those models are evaluated both with and without Routine. But the central claim as worded ('Routine improves tool-calling accuracy') requires an unstructured-plan control to separate format from content. I therefore keep CONDITIONAL, with the added requirement that the authors run this control or temper the causal language. Agreement is partial because the reader's weakest assumption is about evaluation validity, while I focus on the internal comparison validity.","tokens_in":20507,"tokens_out":1224,"duration_ms":13327,"concrete_test":"Re-run the Table 2 comparison with an additional condition: an 'Unstructured Plan' baseline that provides the same step-by-step natural-language instructions and descriptions as the Routine, but omits the explicit 'use tool X' clause from every step, forcing the model to select the tool from its description. If the unstructured-plan accuracy approaches the Routine accuracy (e.g., within 5 points for Qwen3-14B), then the headline gain is primarily due to the plan content rather than the structural format. If the gap remains large, the structured Routine format itself carries the benefit.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Routine improves multi-step tool-calling accuracy. The strongest evidence (Table 2) compares a no-Routine condition, where the model must autonomously choose tools from 25 options, against a Routine condition where each step explicitly states the tool name (e.g., 'use the fetch_latest_announcements tool'). The ablation in §5.3.1 shows that removing the explicit tool name drops Qwen3-14B from 83.3% to 71.9% and Qwen2.5-14B from 79.1% to 69.5%, confirming that the explicit tool name is a major driver of the gain. Thus the headline improvement is not evidence that Routine produces better plans; it largely measures the benefit of converting a tool-selection problem into a tool-name lookup. This is a legitimate engineering contribution, but the paper's framing—'Routine significantly increases execution accuracy'—overstates what is shown. The no-Routine baseline is also disadvantaged because the model receives no plan at all, so the comparison bundles 'has a plan' with 'plan contains the answers.' A fairer baseline would give the model an unstructured natural-language plan with equivalent step content but without explicit tool names, isolating the value of the structured Routine format per se. The reader's concern about self-distilled, oracle-context evaluation is real and worth stating, but the reformulation artifact is more directly load-bearing for the central causal claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Routine, a structured planning framework that provides execution models with step-by-step natural-language plans containing step names, descriptions, and explicit tool assignments, with optional I/O parameter descriptions and branch notation. The framework is evaluated in an enterprise HR agent scenario with 25 MCP tools and seven hand-annotated Routines, using a test set of 1,148 per-step samples decomposed from 200 traces distilled by GPT-4o. Table 2 reports large gains in overall tool-calling accuracy for GPT-4o (41.1% to 96.3%) and Qwen3-14B (32.6% to 83.3%) when Routine is provided, and fine-tuning Qwen3-14B on a common Routine-following dataset plus a scenario-specific distilled dataset raises accuracy to 95.5%. The paper also presents ablations on Routine components, generation methods, and Routine quantity.","tokens_in":20732,"tokens_out":8876,"duration_ms":96961,"significance":"The paper addresses a practical problem—stability of multi-step tool calling in enterprise agents—and proposes a simple, interpretable solution that appears to work across multiple models. The experimental coverage is broad: eight base models, fine-tuning with two data sources, and systematic ablations. The authors are transparent that the explicit tool name is a core component of the framework and that the current approach relies on expert-provided drafts. If the evaluation concerns are resolved, Routine could be a useful practical contribution to applied LLM-agent research. However, the current evaluation methodology (per-step scoring with oracle context, self-distilled test data, and the absence of a control that separates plan presence from plan content) limits the strength of the central claim and requires revision.","major_comments":[{"comment":"The evaluation is per-step and teacher-forced: the 1,148 test samples are decomposed from 200 traces distilled by GPT-4o under the same Routine structure, and each sample is scored with the correct preceding history. Consequently, the overall accuracy measures the model's ability to imitate the teacher's next step given oracle context, not its ability to complete a multi-step task without error propagation. Additionally, the gold labels are the teacher's tool calls without independent verification (e.g., human annotation or ground truth), so the reported accuracy is an agreement rate with GPT-4o, not a measure of task correctness. The authors should either report end-to-end accuracy on complete traces or clearly frame the results as per-step imitation scores.","section":"§5.1.2"},{"comment":"The no-Routine baseline provides no plan, while the Routine condition provides a full plan that includes the correct tool name for every step. The headline gains (41.1→96.3 for GPT-4o) therefore conflate the effect of having a plan with the effect of the plan containing the tool assignments. The ablation in §5.3.1 (Table 3) shows that removing the tool name reduces accuracy but still leaves a structured plan, and the remaining gain over the no-plan baseline indicates that the step descriptions themselves are valuable. To isolate the contribution of the structured format per se, the authors should add a control condition with an unstructured natural-language plan containing the same step descriptions but no explicit tool names, or a structured plan without tool names. Without this control, the claim that the Routine formalism itself drives the improvement is not fully supported.","section":"§5.1.3, §5.3.1"},{"comment":"Both the 537 scenario-specific training queries and the 200 test queries are generated from the same 5–6 query templates per sub-scenario (10 sub-scenarios total). The paper does not state whether the test queries are disjoint from the training queries in terms of template and filled-in entity values, nor does it report the similarity between the two sets. If the test set shares template structure with the training data, the fine-tuned model's 95.5% accuracy may reflect template memorization rather than generalization to new phrasings. The authors should clarify the exact relationship (e.g., new entity values, new template variations) and, ideally, evaluate on a held-out set with novel templates.","section":"§4.2.3, §5.1.2"},{"comment":"The claim that 'tool selection errors were identified as the main cause of failure, accounting for over 85% of all errors' is not directly supported by the table, because the reported tool selection accuracy is computed only on the subset of samples that passed the structural check. Similarly, the statement that the improvement in tool selection accuracy is the main driver of the overall improvement cannot be verified from the conditional metrics alone; different conditions have different subset sizes at each stage. The authors should provide an unconditional error breakdown (e.g., the number or fraction of samples that fail at each stage) to substantiate these conclusions.","section":"Table 2, §5.2.1"}],"minor_comments":[{"comment":"The phrase 'resulting in an accuracy increase to 88.2% on scenario-specific evaluations' is ambiguous; it should specify that this is the Common fine-tuned Qwen3-14B under the 'Routine w/ Branch' condition in Table 2.","section":"Abstract"},{"comment":"'Routing-following dataset' should be 'Routine-following dataset'.","section":"Table 2 caption"},{"comment":"The branch notation in the main text (e.g., 'Branch X-n Step i') differs from the JSON notation in Appendix A.1 (e.g., 'x-n_i'); the authors should state explicitly that these are equivalent representations.","section":"§3.1.1"},{"comment":"The paper states that 537 user queries were obtained after data cleaning, but it is unclear whether the 200 evaluation queries are a subset of these 537 or a separate set; please clarify the relationship.","section":"§4.2.3"},{"comment":"The multi-Routine interference experiment does not specify how the interference Routines are selected (e.g., random, semantically similar, or from different sub-scenarios); this should be stated for reproducibility.","section":"§5.3.3"},{"comment":"'se text parsing tools' appears to be a typo for 'use text parsing tools'.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an industry report without public code or data release. The evaluation relies on proprietary enterprise data and a teacher model whose outputs are not independently verified; if the journal's policy requires reproducibility, the authors should provide an artifact availability statement or make the prompt templates and evaluation code available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering paper with an honest weakness that the authors themselves partially expose. The claim that Routine improves planning is overstated; what the data actually show is that giving the model the correct tool name for each step is what drives the accuracy jump. That is still worth knowing, just not as a planning breakthrough.\n\nThe paper does several things well. It credits the OpenAI Cookbook for the core Routine idea and positions itself as an extension. The ablation in §5.3.1 is the most informative part: removing the explicit tool name drops Qwen3-14B from 83.3% to 71.9%, confirming that the format's main contribution is converting tool selection into a lookup. The controlled comparison (same model, with/without Routine) supports the directional claim, and the fine-tuning experiments on distilled data are a reasonable recipe for enterprise deployment. The paper is transparent about its limitations and does not hide that the evaluation data are self-distilled.\n\nThe soft spots are load-bearing for the causal claim, though not for the engineering recipe. First, the no-Routine baseline gets no plan at all, so the comparison bundles 'has a plan' with 'plan contains the answer.' A fairer baseline would give an unstructured plan with the same step content minus tool names. Second, the evaluation decomposes 200 traces into 1,148 per-step samples scored with oracle context, so 'overall accuracy' is per-step imitation, not end-to-end success. Third, the test set is generated by the same teacher model (GPT-4o) under the same Routine structure being tested; some self-consistency is baked in. No error bars, no code or data release.\n\nNone of this kills the practical value: if you are building an enterprise agent and can hand it explicit tool names, this paper shows you a viable way to do that and how to distill it into a small model. But the abstract's phrasing—'Routine significantly increases execution accuracy'—implies the planning structure itself is the cause. The ablation says otherwise.\n\nWho it's for: practitioners working on LLM tool calling in constrained domains, and anyone designing evaluations for structured prompts. It deserves referee time because the empirical material is real and the ablations are informative, but I would push for a major revision that either reframes the claim or adds the unstructured-plan baseline and reports end-to-end accuracy.\n\nRecommendation: send to review, but expect the authors to substantively revise the causal interpretation.","headline":"Routine is a useful engineering recipe, but the headline gains mostly reflect handing the model the tool names, not better planning.","tokens_in":21337,"tokens_out":2092,"would_cite":false,"duration_ms":22228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A structured planning script lifts an enterprise HR agent's multi-step tool-calling accuracy from 41.1% to 96.3%.","keywords":["Routine","LLM agents","multi-step tool calling","enterprise agent systems","instruction following","planning framework","knowledge distillation","tool selection"],"falsifier":"Re-run the same HR scenario end-to-end without splitting traces: let the agent decide branch conditions and the next step from its own prior tool results, and score each of the 200 queries as a full success only if every tool call and parameter in the whole trajectory is correct. If overall per-step accuracy stays near 96% while full-task success drops substantially, the gain is largely an artifact of oracle step context rather than a real end-to-end capability.","tokens_in":20274,"feed_emoji":"🤖","tokens_out":7246,"duration_ms":582079,"temperature":0.7,"pith_summary":"This paper argues that the reason enterprise LLM agents fail at multi-step tool calling is not raw model capability but plan representation: when a plan is written as a Routine, a structured script with numbered steps, each carrying a name, description, and an assigned tool, the execution model no longer has to infer the workflow. In an HR agent scenario with 25 available tools, giving GPT-4o such a Routine raised its overall per-step accuracy from 41.1% to 96.3%, and gave Qwen3-14B a lift from 32.6% to 83.3%. A reader should care because this converts an expensive, unreliable planning problem into a cheaper instruction-following problem, and the paper shows the same Routine can be used to distill scenario-specific training data that lets a small model reach 95.5%, close to GPT-4o's performance.","feed_headline":"Structured step script lifts agent tool calls from 41% to 96%","feed_subtitle":"Named-step Routine plans turn shaky multi-step tool calls into a followable script, letting small models match GPT-4o.","key_machinery":"The load-bearing object is the Routine itself: a numbered natural-language script in which every step names its purpose, describes what to do, and designates exactly one tool, with branches represented as 'Branch X-n Step i' entries so the execution model can track where it is in the workflow. The complete system wraps the Routine in a system prompt together with a variable-memory dictionary and a standardized tool interface, and a separate procedure-memory mechanism retrieves the one relevant Routine for a query. The routine converts planning into constrained execution: tool selection becomes reading a designated tool name, parameter passing is mediated by short keys that the memory module expands before the tool call, and the final output is generated by a dedicated summarization tool rather than free text from the execution model.","core_discovery":"The central claim is that a Routine—a structured, well-formatted plan composed of step numbers, step names, step descriptions, and explicit tool assignments, plus optional input/output parameter descriptions and branch notation—acts as a stable intermediate representation between a planning model and an execution model. Supplying this Routine in the system prompt turns each tool call from a reasoning and search problem into a direct 'use this tool at this step' instruction. The paper demonstrates the claim in a deployed HR agent scenario: with Routine guidance, GPT-4o's overall accuracy (structure, tool selection, and parameters all correct) rose from 41.1% to 96.3%, and Qwen3-14B rose from 32.6% to 83.3%. Fine-tuning Qwen3-14B on a generic Routine-following dataset improved its routine-guided accuracy to 88.2%, while fine-tuning on 537 scenario-specific tool-calling samples distilled via Routine brought it to 95.5%, approaching GPT-4o's 96.3%.","pith_inferences":["The per-step evaluation gives each sample the correct preceding tool-call history, so the reported 96.3% likely overstates end-to-end task completion; a natural follow-up experiment would compare per-step accuracy with full-trajectory success on the same 200 queries.","If Routine's effect is general, the same annotation-and-evaluation recipe should transfer to other enterprise domains (finance, IT operations, procurement) with a similar 25-tool tool server; the paper's HR-only evidence leaves that transfer open.","Distillation from a routine-annotated teacher suggests a practical deployment path: one domain expert annotates a handful of routines, and a small local model is fine-tuned on distilled traces, reducing dependence on large proprietary models at inference time.","The multi-routine interference results imply retrieval precision matters more than recall for procedure memory; ranking routines with a confidence threshold and returning only the top match may improve stability."],"forward_implications":["Providing a Routine with explicit tool names is the dominant driver of the accuracy gain; removing tool names drops accuracy by roughly 5 to 15 points for most tested models, while adding I/O parameter descriptions helps weaker models and modestly helps some strong ones.","A general Routine-following fine-tuning dataset improves execution accuracy when a Routine is present, but it weakens a model's autonomous planning ability in the no-Routine condition, so the two capabilities trade off.","Scenario-specific distillation through Routines lets a 14B model internalize procedural knowledge and achieve near-GPT-4o accuracy, and adding an explicit Routine on top of the specialized model pushes it even higher.","Branching Routines are slightly harder to follow than linear ones; the accuracy gap is small for strong models but pronounced for weaker ones, so branch logic should be introduced only on top of a solid tool-calling foundation.","When several candidate Routines are present, accuracy can drop because the model may try to merge steps; the paper therefore concludes that the memory module should aim to return a single most relevant Routine."],"supporting_citations":[{"why":"Introduces the basic Routine idea that this paper extends with branches, I/O descriptions, and training data.","marker":"[15]"},{"why":"Supplies the four-module agent architecture (planning, execution, tools, memory) that the proposed system redesigns.","marker":"[2]"},{"why":"Provides the abstract syntax tree evaluation method and error taxonomy used to compute structural, tool-selection, and parameter accuracy.","marker":"[59]"},{"why":"The authors' earlier single-step enterprise tool-calling training pipeline, which this work extends to multi-step scenarios.","marker":"[45]"},{"why":"Establishes the knowledge-distillation basis for using a teacher model to generate scenario-specific tool-calling training data.","marker":"[44]"},{"why":"Supplies the low-rank adaptation method used to fine-tune the small execution models.","marker":"[56]"}],"fun_headline_variants":["Routine framework lifts GPT-4o tool accuracy from 41% to 96%","Structured Routine plans boost tool-call accuracy to 96%","Routine scripts let small models match GPT-4o tool accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claim depends on the test being a fair measure of real task success: each of the 1,148 samples is a per-step slice distilled from GPT-4o traces under the same Routine format being tested, each slice is given the correct preceding tool-call history, and free-text parameters are checked for existence and type rather than exact content.","fun_headline_variants_meta":{"raw":{"variants":["Routine framework lifts GPT-4o tool accuracy from 41% to 96%","Structured Routine plans boost tool-call accuracy to 96%","Routine scripts let small models match GPT-4o tool accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2447,"prompt_tokens":1032,"completion_tokens":1415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1351}},"tokens_in":648,"tokens_out":1415,"duration_ms":11221,"temperature":1.0,"reasoning_tokens":1351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:55:33.135885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same HR scenario end-to-end without splitting traces: let the agent decide branch conditions and the next step from its own prior tool results, and score each of the 200 queries as a full success only if every tool call and parameter in the whole trajectory is correct. If overall per-step accuracy stays near 96% while full-task success drops substantially, the gain is largely an artifact of oracle step context rather than a real end-to-end capability.","supporting_citations":[{"cited_title":"Using reasoning for routine generation","cited_arxiv_id":null,"evidence_quote":"Introduces the basic Routine idea that this paper extends with branches, I/O descriptions, and training data."},{"cited_title":"Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E","cited_arxiv_id":null,"evidence_quote":"Provides the abstract syntax tree evaluation method and error taxonomy used to compute structural, tool-selection, and parameter accuracy."},{"cited_title":"Adaptable and precise: Enterprise- scenario llm function-calling capability training pipeline, 2024","cited_arxiv_id":null,"evidence_quote":"The authors' earlier single-step enterprise tool-calling training pipeline, which this work extends to multi-step scenarios."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Lu Wang","cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank adaptation method used to fine-tune the small execution models."}],"review_version":1}