{"id":"e6931c77-d341-497e-9706-407434763891","arxiv_id":"2505.09970","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pre-Act adds explicit multi-step planning to ReAct agent prompts; a fine-tuned 70B Llama reached 0.92 action recall on Almita, beating GPT-4's 0.54.","lead":"The paper introduces Pre-Act, which makes LLM agents write a multi-step plan with reasoning before each action and revise it after each tool result. This could let smaller, cheaper models replace larger APIs like GPT-4 in customer-service agent tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuned gains conflate Pre-Act with proprietary-data fine-tuning; a ReAct fine-tuned control is missing, so the headline 69.5% turn-level and 28% E2E gains cannot yet be attributed to Pre-Act.","rationale":"Both the Reader and I see the E2E protocol as fragile, and I agree with the CONDITIONAL verdict. My primary reason is not the judge itself but a missing experimental control that undermines attribution of the fine-tuned model's gains to Pre-Act. The Stage-2 training uses a proprietary Pre-Act-formatted dataset with human-written reasoning; no Stage-2 ReAct-formatted control exists. Thus the 69.5% turn-level gain and the 28% E2E gain could be due to the data and fine-tuning, not the planning format. This concern is checkable: fine-tune the same base model on the same two-stage curriculum with the second stage in ReAct format. If the control matches Pre-Act's numbers, the central mechanism claim fails; if it does not, Pre-Act's contribution is confirmed. The vanilla-model comparisons in Table 2 do give real, independent support for Pre-Act as a prompting method on pretrained models, so I would not reject the paper; I would require the control experiment, plus error bars or a human-judge validation for the E2E numbers, before accepting the headline causal claims. The Reader's weakest assumption about LLM-as-a-judge validity is valid but secondary; the missing ReAct baseline affects both turn-level and end-to-end fine-tuned results and is therefore the more fundamental attribution gap.","tokens_in":14684,"tokens_out":9867,"duration_ms":95275,"concrete_test":"Run a control fine-tune of Llama-3.1-70B through the identical two-stage curriculum (Glaive ReAct, then the same proprietary dataset), but format the second-stage examples as standard ReAct (single-step thought plus action, no multi-step plan). Evaluate this control on Almita turn-level and on the same five-use-case E2E setup, reporting per-use-case variance. If its action recall and goal completion match or approach the Pre-Act model's 0.9238 and 0.82, the reported advantage is a fine-tuning or data effect rather than an effect of Pre-Act; if it falls well short, the Pre-Act attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 presents a two-stage curriculum: Stage 1 fine-tunes Llama models on Glaive in ReAct format; Stage 2 continues on a proprietary dataset in Pre-Act format. Table 4 only measures Stage-1 vs Stage-2 retention on Glaive. No model is ever fine-tuned on the same proprietary data in ReAct format. Consequently, Table 2's fine-tuned rows and Table 3's E2E rows compare the Pre-Act fine-tuned system against vanilla models and GPT-4, not against an equivalent ReAct fine-tuned system. The 69.5% Almita action-recall gain and the 0.82 goal-completion rate could therefore be produced by the additional high-quality proprietary annotations and QLoRA fine-tuning alone, independent of Pre-Act's multi-step plan. This is a missing control, not an internal inconsistency, but it is load-bearing because the abstract attributes the gains to Pre-Act. The E2E judge protocol (Appendices F/H) adds a second-order validity risk, since GPT-4 both defines milestones and scores them, and the conclusion concedes 'volatility in LLM-as-a-judge assessments'; the missing ReAct control, however, affects the turn-level result too.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Pre-Act, a modification of ReAct in which the agent generates a multi-step execution plan with per-step reasoning before each action, refining the plan as tool observations arrive. The authors evaluate Pre-Act as a prompting strategy on five pretrained models (Llama 3.1 8B/70B, Nemotron 70B, DeepSeek-distill 70B, GPT-4) on three datasets (Glaive, proprietary, Almita) using turn-level metrics (action recall, F1, parameter match, similarity), and as a fine-tuning target for Llama 3.1 8B/70B using a two-stage curriculum (Glaive in ReAct format, then proprietary data in Pre-Act format). They also propose an end-to-end evaluation that uses GPT-4 to build milestone dependency graphs and GPT-4 as a judge to measure goal completion and progress rate on five Almita use-cases. The headline claims are that Pre-Act improves turn-level action recall over ReAct by 70% on average (Almita) and that a fine-tuned 70B model outperforms GPT-4 by 69.5% in turn-level action accuracy and by 28% in end-to-end goal completion rate on Almita.","tokens_in":14956,"tokens_out":5055,"duration_ms":51490,"significance":"If the results hold, Pre-Act is a simple, broadly applicable prompting and fine-tuning scheme that could improve agent performance across model sizes. The turn-level comparisons in Table 2 are internally consistent, the authors provide code and dataset contributions for Almita, and the curriculum-learning retention analysis in Table 4 is a useful check. However, the central claims for the fine-tuned models rely on a missing control: no model is fine-tuned on the same proprietary data in ReAct format, so the reported gains over GPT-4 and vanilla models cannot be attributed to multi-step planning alone. The end-to-end evaluation is a second-order validity risk because GPT-4 generates the milestones and also serves as the judge, and the paper itself concedes judge volatility. The significance of the paper is therefore conditional on whether the authors can disentangle Pre-Act from the effect of additional domain-specific fine-tuning.","major_comments":[{"comment":"The attribution of the fine-tuned gains to Pre-Act is not supported by the experimental design. The two-stage curriculum fine-tunes on Glaive in ReAct format (Stage 1) and then on proprietary data in Pre-Act format (Stage 2), but no model is ever fine-tuned on the same proprietary data in ReAct format. The only curriculum control, Table 4, measures retention on Glaive across stages, not the effect of Pre-Act versus ReAct. Consequently, the 69.5% turn-level action-accuracy improvement and the 0.82 goal-completion rate reported for the fine-tuned 70B model in Section 5 could be entirely produced by the additional high-quality proprietary annotations and QLoRA fine-tuning, independent of the multi-step plan. The authors should add a Stage-2 control fine-tuned on the proprietary dataset formatted as ReAct, with the same data split, training hyperparameters, and evaluation protocol, and report the comparison in Tables 2 and 3.","section":"§3.2, Tables 2–4"},{"comment":"The end-to-end evaluation is a closed loop: GPT-4 generates the milestone dependency graphs (Appendix F), a GPT-4-based synthetic user simulates the conversation, and GPT-4 as judge scores milestone completion (Appendix H). The paper's conclusion concedes 'volatility in LLM-as-a-judge assessments,' yet no agreement study against human judges, no alternative judge, and no per-run variance are reported. Without such evidence, the 28% goal-completion gain of the fine-tuned model over GPT-4 Pre-Act could be an artifact of the judge favoring responses whose style resembles the model's own milestone descriptions. Please report the distribution and confidence intervals over the 50 simulated runs, and provide a human-annotation agreement analysis or an alternative judge (e.g., a different LLM, or rule-based verification against tool outputs) to establish that the milestone-completion metric measures real task success.","section":"§4.3.1, Appendices F/H"},{"comment":"Several headline comparisons lack statistical support. All turn-level metrics are single runs without multiple seeds or error bars, and the E2E averages are reported without variances over the 50 runs. For claims of 70%, 128%, and 69.5% relative improvements on datasets with only a few hundred test instances (Table 1: 378 proprietary, 1100 Almita), the authors should provide bootstrap confidence intervals or significance tests. Otherwise the reader cannot judge whether the differences exceed run-to-run variability. This is particularly important because the fine-tuned rows in Table 2 are compared against prompted baselines rather than equally trained alternatives, as noted in the first comment.","section":"§5, Tables 2–3"},{"comment":"The framing that the fine-tuned 70B model 'outperforms GPT-4' overstates the scientific comparison: it juxtaposes a model fine-tuned on task-specific proprietary data (plus the Almita annotations created by the same team) with a prompted generalist. This comparison is legitimate as an engineering benchmark but does not isolate Pre-Act as the cause of the improvement. The abstract and conclusion should be reworded to distinguish the effect of the Pre-Act prompting strategy (supported by the vanilla-model rows in Table 2) from the combined effect of fine-tuning and the Pre-Act format, unless the ReAct fine-tuned control from the first comment demonstrates that the fine-tuning benefit is specific to Pre-Act.","section":"§5"}],"minor_comments":[{"comment":"In the Pre-Act example, the observation text says 'get_my_headlines' while the tool is 'get_news_headlines'; this typo should be fixed.","section":"Figure 1"},{"comment":"The entries '-' for vanilla-model Pre-Act on Glaive are not explained in the caption; please state explicitly that Pre-Act annotations are unavailable for Glaive, so those comparisons are not performed.","section":"Table 2 caption"},{"comment":"Training details are incomplete: please report the LoRA rank, alpha, target modules, quantization settings, number of GPUs, total runtime, and any distributed training setup for the 70B model, as these are needed for reproducibility.","section":"Appendix A"},{"comment":"The text says 'LoRA training' in one place and 'QLoRA' in Appendix A; please clarify whether quantized base models were used and specify the exact configuration.","section":"§3.2"},{"comment":"The milestone dependency graph is described as generated in YAML format, but no machine-readable example is provided; Figure 6 is an image. Please include the actual YAML used for the Almita use-cases.","section":"Appendix F/G"},{"comment":"The statement that Pre-Act improves action recall by 70% on average 'across five models' should specify that this is the mean of per-model relative improvements, since Table 2 shows the relative gains vary from roughly 23% to 168% across models.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The missing ReAct fine-tuned control is the central technical issue and is fixable within the paper's scope. The end-to-end judge loop is a second major concern; the authors should provide at least a human-annotation agreement check or variance reporting before the goal-completion claims are acceptable. The paper is from an industrial lab and relies on a proprietary dataset, which limits external reproducibility, but the Almita contribution and code release partially mitigate this. The abstract's claim of outperforming GPT-4 is likely to attract attention, so the authors should be held to a high standard for causal attribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: Pre-Act is a modest but sensible extension of ReAct—generate an explicit multi-step plan plus per-step reasoning before each action, refine as tool observations come in. On vanilla models the turn-level comparisons are internally consistent and the improvement is real. The two-level evaluation (turn-level plus end-to-end with milestone graphs) is a nice contribution, and the curriculum fine-tuning that checks for catastrophic forgetting on Glaive is good practice. The paper is worth reading for the evaluation design and the fine-tuning recipe.\n\nThe soft spots are where the abstract goes furthest. The 69.5% and 28% gains for the fine-tuned 70B are attributed to Pre-Act, but there is no control where the same proprietary data and QLoRA fine-tuning are applied in plain ReAct format. Stage 1 fine-tunes on Glaive with ReAct; Stage 2 fine-tunes on the proprietary dataset with Pre-Act. Without a ReAct fine-tuned baseline trained on the same proprietary data, you cannot separate the effect of Pre-Act's planning from the effect of adding high-quality proprietary annotations and further fine-tuning. That is load-bearing, because the abstract names Pre-Act as the cause. The stress-test note is correct: this is a missing control, not an inconsistency, but it means the headline claim is not yet supported.\n\nThe end-to-end evaluation also rests on GPT-4 for both milestone generation (human-verified) and judging, and the authors themselves concede volatility in LLM-as-a-judge. That makes the E2E numbers fragile. Minor issues: no error bars, no comparison to Plan-and-Solve or Least-to-Most, and the proprietary training data is not released, so reproducing the fine-tuning results is impossible from the paper alone.\n\nIf this were mine to decide, I would send it to peer review with a request for a ReAct fine-tuned control on the same proprietary data, and ideally a different judge or human evaluation for the E2E part. The vanilla-model results and the evaluation framework are enough to justify a serious referee.\n\nRegards,","headline":"Useful ReAct variant with honest vanilla-model results, but the fine-tuning headline is not yet supported: no ReAct fine-tuned control, so the gains may come from the proprietary data and QLoRA, not from Pre-Act.","tokens_in":15473,"tokens_out":1742,"would_cite":false,"duration_ms":17040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM agent which writes and revises a multi-step plan before each action gets far better tool-use accuracy, and that a 70B model fine-tuned on such plans beats GPT-4 on out-of-domain agent tasks.","keywords":["LLM agents","multi-step planning","ReAct","tool calling","curriculum fine-tuning","goal completion","LLM-as-a-judge","milestone graph"],"falsifier":"Re-score the end-to-end Almita conversations with human annotators instead of the GPT-4 judge, comparing milestone completion against the same human-verified graphs; if human scores do not reproduce the reported 0.82 versus 0.64 goal-completion gap, the end-to-end claim is an evaluation artifact.","tokens_in":14528,"feed_emoji":"🤖","tokens_out":9253,"duration_ms":85374,"temperature":0.7,"pith_summary":"The paper's central claim is that an LLM agent acts more reliably when, before each action, it writes out a multi-step plan of the remaining actions and revises that plan after every tool result, a method the authors call Pre-Act. Against ReAct, which reasons only about the immediate next action, Pre-Act reports a 70% average improvement in action recall across five models on the Almita dataset and a 102% improvement on the authors' proprietary dataset. The paper further claims that small models fine-tuned in two stages on Pre-Act traces can match or beat much larger proprietary ones: the fine-tuned Llama 3.1 70B beats GPT-4 by 69.5% in turn-level action accuracy and 28% in end-to-end goal completion on out-of-domain Almita. A reader should care because the result points to plan structure, not model scale, as a lever for agent reliability, and because the paper pairs it with a two-level evaluation that separates correct tool calls from actually finishing the task.","feed_headline":"Plan-first agents beat ReAct and let a 70B model top GPT-4","feed_subtitle":"Fine-tuned on multi-step plan traces, Llama 3.1 70B lifts action recall by 69.5% on out-of-domain tasks","key_machinery":"The load-bearing object is the multi-step plan written inside the model's internal 'Thought'. Pre-Act defines a structured format: at each turn the model must produce Previous Steps, a summary of executed actions and observations, and Next Steps, a numbered plan with detailed reasoning for each upcoming action ending with the final answer. The plan is regenerated after each observation, so the model is always committing to the remaining trajectory rather than only the next tool call. Training uses curriculum fine-tuning: stage one teaches tool calling on a large ReAct-formatted dataset, and stage two uses expert-annotated Pre-Act traces on a small proprietary dataset with parameter-efficient LoRA updates to preserve stage-one skills. Evaluation's key object is the milestone dependency graph, generated by GPT-4, human-verified, with functional milestones tied to tool names and non-functional milestones tied to workflow states; goal completion is measured by how far along this graph the agent's actions are judged to have advanced.","core_discovery":"On its own terms, the paper's discovery is that planning should come before acting and should be revised after every observation. Whereas ReAct generates a thought tied to the immediate action, Pre-Act makes the model commit to the full remaining trajectory at each step: it writes the previous steps already taken and a numbered plan of next steps, executes the next tool call, folds the observation into the plan, and repeats. The paper reports that this change alone raises action recall over ReAct by 102% on its proprietary dataset and 70% on Almita, averaged across five models. It then shows that small models trained in two stages—first to imitate ReAct tool-calling at scale, then to imitate expert-annotated Pre-Act plans on a small dataset—retain the tool-calling skill and gain the planning skill; the fine-tuned 70B model reaches 0.9238 action recall on Almita and 0.82 average goal completion across five end-to-end use-cases, compared to 0.5449 and 0.64 for GPT-4 with Pre-Act.","pith_inferences":["Editorial inference: the end-to-end gains should be read with the judge in the loop, because the milestone graphs and the judge are both GPT-4; part of the 28% gap may reflect the fine-tuned model producing traces that match the judge's preferred format, and an independent human re-scoring would settle this.","Editorial inference: because Pre-Act's gains appear on out-of-domain tools, the mechanism may be transferable planning behavior rather than memorized tool schemas; a testable extension is to freeze the planner and vary unseen tools to see whether plan quality predicts action accuracy.","Editorial inference: the planning format could be combined with inference-time strategies such as sampling multiple plans or backtracking on failed observations, which the paper does not explore; if plan regeneration is what drives recovery, those combinations should compound.","Editorial inference: if planning before action is the active ingredient, then forcing a plan after choosing an action should not help; a simple ablation that swaps the order of plan and action at inference would test whether lookahead causes the gain or merely correlates with it."],"forward_implications":["In setups similar to the five tested, replacing ReAct's single-step thought with a regenerated multi-step plan should raise the chance that the right tool is called; the averaged action-recall gain reported is 70% on Almita.","Fine-tuning a 70B model on Pre-Act traces is enough to beat a much larger proprietary model in this task family: 69.5% higher turn-level action accuracy and 28% higher end-to-end goal completion on Almita, with lower latency and cost.","The two-level evaluation separates tool-call correctness from task completion, so a system that picks the right tools but never finishes the workflow will no longer look fully successful.","A curriculum that first teaches tool calling and then refines on expert-annotated plan traces avoids the usual forgetting: Glaive action recall drops only 0.80 points for the 8B model and 0.36 for the 70B after stage two.","The same Pre-Act prompting also helps GPT-4 in end-to-end settings, raising its average goal completion from 0.32 with ReAct to 0.64 with Pre-Act."],"supporting_citations":[{"why":"Supplies the ReAct thought-action-observation loop that Pre-Act modifies and the baseline it must beat.","marker":"Yao et al., 2023b"},{"why":"Introduces milestones-based progress and goal-completion metrics that the end-to-end evaluation adapts.","marker":"Gioacchini et al., 2024"},{"why":"Provides the LLM-as-a-judge paradigm used to score which milestones the agent achieved.","marker":"Zheng et al., 2023"},{"why":"Provides QLoRA, the parameter-efficient fine-tuning method used in curriculum training.","marker":"Dettmers et al., 2023"},{"why":"Provides curriculum learning, the two-stage training schedule the paper follows.","marker":"Bengio et al., 2009"},{"why":"Introduces synthetic-user simulation, the basis of the end-to-end evaluation environment.","marker":"Levin et al., 2000"}],"fun_headline_variants":["Plan-before-act boosts LLM agents, 70B model tops GPT-4","Pre-Act: plan and refine to beat ReAct in agent tasks","Fine-tuned 70B Llama surpasses GPT-4 with Pre-Act planning","Multi-step planning lifts agent recall 70% over ReAct","Small model fine-tuned with Pre-Act outperforms GPT-4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The end-to-end result rests on the assumption that GPT-4's milestone-scoring, human-checked only at graph-construction time, agrees with what a human would count as task completion; the paper itself concedes 'volatility in LLM-as-a-judge assessments' in its conclusion.","fun_headline_variants_meta":{"raw":{"variants":["Plan-before-act boosts LLM agents, 70B model tops GPT-4","Pre-Act: plan and refine to beat ReAct in agent tasks","Fine-tuned 70B Llama surpasses GPT-4 with Pre-Act planning","Multi-step planning lifts agent recall 70% over ReAct","Small model fine-tuned with Pre-Act outperforms GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2687,"prompt_tokens":1105,"completion_tokens":1582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":1481}},"tokens_in":721,"tokens_out":1582,"duration_ms":11702,"temperature":1.0,"reasoning_tokens":1481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:19:20.277041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the end-to-end Almita conversations with human annotators instead of the GPT-4 judge, comparing milestone completion against the same human-verified graphs; if human scores do not reproduce the reported 0.82 versus 0.64 goal-completion gap, the end-to-end claim is an evaluation artifact.","supporting_citations":[],"review_version":1}