{"id":"8cd3a987-d11d-4767-a14a-d347a57d6266","arxiv_id":"2504.16563","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GoalAct, a framework that pairs continuously updated global planning with hierarchical skills, reports a 12.22% average success-rate gain over four agent baselines on LegalAgentBench.","lead":"This paper proposes GoalAct, an LLM agent framework that keeps a global plan updated while executing tasks through high-level skills such as searching, coding, and writing. On the LegalAgentBench legal-domain benchmark, GoalAct reports an average 12.22% success-rate improvement over prior agent methods across three LLMs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's substring success metric may inflate GoalAct's 12.22% gain; a blind human-scored sample of outputs would settle whether the benchmark measures real task completion.","rationale":"Read in good faith, GoalAct is a coherent framework: a continuously updated global plan plus hierarchical skills, with consistent gains over four baselines on three LLMs and an ablation on GLM-4-Plus. The framework itself is plausible and the reported numbers are internally consistent with Table 2. The single most load-bearing weakness is not the architecture but the evaluation instrument. The success metric in Eq. 4 counts substring occurrences of short key answers; under this metric, methods that emit longer, more document-shaped outputs, such as GoalAct's Writing skill, can score higher without demonstrating better legal reasoning or document quality. This is a measurable correctness risk, not a charge of bad faith: even a well-intentioned same-lab benchmark can be inadvertently aligned with the method's action vocabulary. The authors report no variance or significance testing, so the SOTA claim rests on point estimates from a single run at temperature 0. The prescribed human-scoring check directly targets the metric; if it passes, the central claim is substantially supported. Given this, the reader's CONDITIONAL verdict is appropriate; I would not reject, but I would require the metric validation or an external benchmark before calling the result a settled state of the art.","tokens_in":9357,"tokens_out":4013,"duration_ms":40003,"concrete_test":"Randomly sample 50 Writing and 50 multi-hop outputs from GoalAct and from the strongest baseline (ReAct or CodeAct) on GLM-4-Plus. Have two legal experts, blind to method, score each output for actual task completion: correct legal entities, correct reasoning chain, and a usable written document. Compare the expert ranking with the ranking produced by Eq. 4. If expert scores do not reproduce GoalAct's roughly 12% advantage, the success-rate metric is the source of the headline gain and the SOTA claim should be re-qualified as a higher substring-match rate on a same-lab benchmark. If expert scores do reproduce the advantage, the concern is resolved and the conditional verdict can be upgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that GoalAct beats four baselines by 12.22% average success rate on LegalAgentBench. All reported numbers come from Eq. 4, which scores each output by the fraction of key answer strings that appear as substrings of the model output. In Writing tasks, each query has roughly 10.25 key answers with average string length 12.58, and LegalAgentBench's writing items are free-form defense documents. GoalAct's Writing skill instructs the model to produce such documents directly, so its outputs are long natural-language texts in which short legal phrases from the answer key can appear by substring matching even when the document is legally weak. ReAct and CodeAct have no equivalent writing branch, and CodeAct in particular performs poorly on Writing (0.42-0.65 vs GoalAct 0.80-0.86). Thus the metric may be rewarding output length and format rather than task completion, and the claimed 12.74% writing improvement may not reflect legal quality. The same mechanism can affect multi-hop tasks, where short key answers may appear incidentally in longer trajectories. Since LegalAgentBench was created by an overlapping author group and its task types mirror GoalAct's hand-selected skills (searching, coding, writing), benchmark independence does not mitigate this risk. No error bars or repeated runs are reported, so the smaller per-task margins cannot be assessed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GoalAct, an LLM-agent framework that combines a continuously updated global plan with hierarchical execution decomposed into high-level skills (Searching, Coding, Writing, Finish). The method is evaluated on LegalAgentBench, a 300-task legal-domain benchmark with 1-hop to 5-hop reasoning tasks and a writing task, using three LLMs (GPT-4o-mini, Qwen-max, GLM-4-Plus) and four baselines (Plan-and-Solve, Plan-and-Execute, ReAct, CodeAct). The central claim is state-of-the-art performance, with an average 12.22% improvement in success rate over the best baseline, driven by stronger performance on multi-hop and writing tasks. An ablation on GLM-4-Plus shows degradation when the global plan or individual skills are removed. The authors release code at a public repository.","tokens_in":9598,"tokens_out":2359,"duration_ms":24632,"significance":"If the reported gains are valid, GoalAct offers a simple and modular recipe for improving LLM agents on tool-using tasks: separate high-level skill selection from low-level tool invocation, and maintain a global plan that is updated at every step. The experimental design has useful strengths: it tests three LLM families, sets temperature to 0 for reproducibility, includes a benchmark designed to avoid data leakage, and provides open-source code. However, the evidential basis for the headline SOTA claim is currently limited by the evaluation metric, the lack of statistical reliability analysis, and the close relationship between the benchmark's task types and the method's hand-selected skill set. The contribution is therefore promising but not yet established at the level claimed.","major_comments":[{"comment":"The success-rate metric counts the fraction of key answer strings that appear as substrings of the model output. This is a weak proxy for task completion, especially for Writing tasks, where the average key answer length is only 12.58 characters and GoalAct explicitly instructs the model to generate a formal defense document. Longer, format-compliant outputs can mechanically include such short legal phrases without being legally adequate, so the reported 12.74% Writing improvement (Table 2) may overstate real gains. The paper should either use a stricter exact-match or semantic-match metric, or report a human-scored sample of outputs, to establish that the improvement reflects task completion rather than output length or format.","section":"§4.1.3, Eq. (4)"},{"comment":"No error bars, confidence intervals, or significance tests are reported, and there is no indication of repeated runs or multiple random seeds. Given temperature 0, run-to-run variance may be low, but tool-execution outcomes and API nondeterminism can still vary. Several per-cell margins are small (e.g., GPT-4o-mini 1-hop: GoalAct 0.9556 vs ReAct 0.9333, and 4-hop: 0.5625 vs CodeAct 0.3875). Without variance estimates or paired significance tests, the claim that GoalAct consistently achieves SOTA \"across different LLM series and varying task difficulties\" is not statistically grounded. The authors should report standard deviations over at least several runs and apply a paired test (e.g., bootstrap or Wilcoxon) to the per-task success scores.","section":"§4.2.1, Table 2"},{"comment":"LegalAgentBench was created by an overlapping author team (reference [12]) and its task types—searching over legal databases, coding-style multi-hop queries, and writing defense documents—mirror the exact skill set hard-coded in GoalAct (Searching, Coding, Writing, Finish). This creates a risk that the benchmark is implicitly shaped toward the proposed method's action space, so the reported 12.22% average improvement may not transfer to other agent benchmarks or to general legal work. The paper should evaluate GoalAct on at least one independent agent benchmark (e.g., AgentBench or a legal-domain benchmark with different task structures) or explicitly discuss why the overlap does not inflate the results.","section":"§4.1.1 and §3.2"},{"comment":"The ablation is conducted only on GLM-4-Plus, and the paper does not report whether the removal of the global plan or of individual skills is accompanied by any prompt re-optimization or budget adjustment. Since the maximum iteration limit T=10 is fixed, removing a skill changes how the limited budget is spent; for example, \"w/o writing\" may force the model to use Searching or Coding for a writing task, which could explain part of the 4.46% overall drop. Repeating the ablation on at least one additional LLM and reporting the distribution of outcomes across runs would make the component-wise attribution more convincing.","section":"§4.2.2, Table 3"}],"minor_comments":[{"comment":"The introduction mentions \"reasoning\" as a potential high-level skill, and Section 3.2 says reasoning tasks may require specialized skills, but the implemented action space in Table 4 only includes Searching, Coding, Writing, and Finish. The paper should clarify whether reasoning is an actual skill in the evaluated system or only a planned extension.","section":"§3.2"},{"comment":"The description of Plan-and-Solve and Plan-and-Execute baselines is brief; it does not specify whether they use the same tool set, the same two in-context examples, or the same maximum iteration limit T=10. Making these details explicit would strengthen reproducibility.","section":"§4.1.2"},{"comment":"The claim that LegalAgentBench \"eliminates the risk of data leakage\" is reasonable for a newly released benchmark, but the paper should also note that the benchmark is from the same research group as this work, which is relevant to assessing benchmark independence.","section":"§4.1.1"},{"comment":"The case study would be more informative if the full GoalAct trajectory (including the global plan updates after each observation) were shown. The current excerpt only shows the first few steps, making it hard to verify the claim that global planning prevents repeated local-branch attempts.","section":"Figure 2"},{"comment":"The notation G=(P1A1,P2A2,...) is used before the plan-update rule is introduced, and the subscript indices in Equation (2) are not fully defined (what exactly is T in G_t = π(Q|T|S_t)?). Clarifying the notation would improve readability.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is empirical and currently rests on a substring-match metric and a benchmark from the same lab. If the metric is validated by human evaluation and the statistical analysis is added, the contribution could be solid. I would not reject outright, but the SOTA claim as stated is not yet supported. Also note that the code repository, while a positive, should be checked for the exact evaluation harness to confirm metric handling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does what it says—builds a simple, sensible agent framework that beats four baselines on LegalAgentBench by a consistent margin. The 12.22% headline is believable as an in-benchmark result, but I would not treat it as a settled statement about real legal work until the metric and benchmark independence are addressed.\n\nWhat's genuinely new here is the combination: a continuously updated global plan plus a small set of high-level skills (searching, coding, writing). That's a natural extension of ReAct and CodeAct, and the paper shows it helps across three LLMs and most difficulty levels. The ablation on GLM-4-Plus is informative—coding removal hurts the most, writing removal least—and releasing code is a plus. The writing is clear; equations (1)–(4) are just definitions, which is fine for an architecture paper.\n\nThe soft spots are real but addressable. First, there are no error bars or significance tests. Some per-task margins are small, and with three LLMs and multiple methods, a few numbers could easily flip. Second, LegalAgentBench was built by an overlapping author team, and the skill set (searching/coding/writing) was hand-picked for that benchmark's task types. That does not make the result wrong, but it does mean the benchmark is not independent evidence for the framework's generality. Third, the success metric is substring matching (Eq. 4). In Writing tasks, the model produces long defense documents while the answer key contains short phrases (~12 chars on average). GoalAct's Writing skill instructs the model to produce such documents, so its outputs are longer and more likely to contain key phrases by chance—or at least by format rather than by legal quality. CodeAct, which performs poorly on writing, has no equivalent branch. I think the stress-test note is on target: a blind human-scored sample of outputs would settle this. The absence of statistical tests is minor in context, but the metric concern is the load-bearing issue.\n\nWho is this for? Researchers building LLM agents for tool-heavy tasks, and anyone thinking about how to evaluate them honestly. The paper deserves a serious peer review, not a desk reject—but the reviewers should ask for repeated runs, a second benchmark or human evaluation, and explicit disclosure of the benchmark overlap.","headline":"A plausible 12.22% gain on LegalAgentBench, but the substring-matching metric and same-lab benchmark mean the real-world transfer is unproven; deserves peer review with revisions.","tokens_in":10151,"tokens_out":2580,"would_cite":false,"duration_ms":25633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GoalAct, a framework coupling continuously updated global planning with hierarchical skill execution, reports state-of-the-art performance on LegalAgentBench, improving average success rate by 12.22% over existing agent baselines.","keywords":["large language model","agent","global planning","hierarchical execution","legal domain","tool use","benchmark evaluation","success rate"],"falsifier":"Run GoalAct and the strongest baseline on a fresh set of legal tasks authored independently of the benchmark team, scoring by exact answer matching and by whether the final answer would satisfy a legal expert, and check whether the 12.22% average improvement persists; if the margin shrinks to zero or reverses, the central claim fails.","tokens_in":9110,"feed_emoji":"⚖️","tokens_out":9330,"duration_ms":76184,"temperature":0.7,"pith_summary":"This paper argues that LLM-based agents fail on complex tool-using tasks because planning and execution are separated and because actions are treated as a single flat level, forcing a choice between narrow action formats and fragile complex ones. It proposes GoalAct, which keeps a global plan that is regenerated after every action from the query, the available tools, and the execution history, and which decomposes execution into high-level skills—searching, coding, writing, and finishing—so the plan only has to choose a skill and an objective. The authors evaluate GoalAct on LegalAgentBench, a 300-task legal benchmark requiring external tools and legal knowledge, and report that it outperforms existing agent frameworks on all three tested LLMs, with an average success-rate improvement of 12.22%. If correct, this shows that coupling a continuously updated global plan with hierarchical skill execution reliably improves LLM agents on multi-step tool-use tasks, and that the improvement grows as task difficulty increases.","feed_headline":"GoalAct boosts LLM agent success by 12.22% on legal tasks","feed_subtitle":"A continually updated plan plus search, code, and write skills beats ReAct and CodeAct on 300 legal tool-use tasks.","key_machinery":"The central machinery is the continuously updated global plan together with hierarchical skill decomposition. The global plan is a sequence of plan-step/action pairs; at each time step the update policy $\\pi$ regenerates the whole plan from the user query $Q$, the available tool set $T$, and the execution history $S_t$, so observations feed back directly into the next plan. Each action in the plan is one of a small set of high-level skills—Searching for text/json tool calls, Coding for Python code with loops and branches, Writing for document generation, and Finish—and the skill itself handles the low-level tool selection and parameter setting. This separation lets the plan stay abstract enough to remain feasible while concrete enough to guide execution, and it lets the skill set grow without redesigning the planner.","core_discovery":"The central claim is that the main failure modes of LLM agents—getting trapped in local branches, producing plans that cannot actually be executed, and using an action space that is either too narrow or too error-prone—can be overcome by two coordinated design choices. First, GoalAct maintains a global plan $G = (P_1A_1, P_2A_2, \\ldots, P_nA_n)$ and regenerates this plan at every step with an update policy $\\pi$ that takes the user query $Q$, the tool set $T$, and the execution history $S_t$ as input, so the plan always incorporates observations from actions already taken. Second, execution is organized hierarchically: the plan step names a high-level skill—Searching (ReAct-style text or JSON tool calls), Coding (Python code with loops and branches), Writing (document generation), or Finish—and that skill then handles tool selection and parameter details. On LegalAgentBench, GoalAct reports average success-rate improvements of 14.45% for GPT-4o-mini, 10.09% for Qwen-max, and 12.11% for GLM-4-Plus over the best baseline, and the ablation study shows that removing the global plan or any of the three skills lowers performance, with removing coding costing the most (14.06% average).","pith_inferences":["Beyond the paper, the hierarchical decomposition suggests a modular route to extending agents to new domains: adding a new high-level skill should require no change to the planning mechanism, a claim the paper motivates but does not test outside legal tasks.","Beyond the paper, the every-step regeneration of the global plan could be made more cost-efficient by updating the plan only when an observation contradicts the current path; replanning every step is the design choice here, and its computational cost is not measured in the paper.","Beyond the paper, the success metric's reliance on key-answer string occurrence leaves open the possibility that an agent could score partial success without producing a usable final artifact; a stricter outcome-based evaluation would test whether the 12.22% gain corresponds to genuinely useful legal outputs."],"forward_implications":["GoalAct reports the highest success rate on LegalAgentBench for every base LLM tested, with an average 12.22% improvement over the second-best method (14.45% for GPT-4o-mini, 10.09% for Qwen-max, 12.11% for GLM-4-Plus).","The advantage widens with task difficulty: relative improvements average 3.26% on 1-hop tasks, 20% on 3-hop tasks, and 15.5% on 5-hop tasks, so the framework is most valuable exactly where agents usually break.","Ablations show each component carries weight: removing the global plan costs 8.14% average success, removing coding costs 14.06%, and removing the writing skill lowers writing-task success by 3.96%.","Because the plan only names high-level skills, the skill set can be extended without redesigning the planner, which the paper presents as the route to broader applicability beyond legal tasks."],"supporting_citations":[{"why":"Defines the Plan-and-Solve baseline that executes a static plan without updating, against which GoalAct shows the value of dynamic replanning.","marker":"[6]"},{"why":"Supplies the ReAct thought-action-observation baseline and the text/json tool-calling format that GoalAct's Searching skill inherits.","marker":"[7]"},{"why":"Supplies the CodeAct baseline whose Python action space motivates the Coding skill and the reported instability of multi-tool code.","marker":"[8]"},{"why":"Provides the Plan-and-Execute baseline that dynamically updates plans but lacks concrete executable actions.","marker":"[9]"},{"why":"Introduces LegalAgentBench, the 300-task legal benchmark that provides all tasks, tools, and the success-rate metric used in the evaluation.","marker":"[12]"},{"why":"Documents GPT-4o-mini, one of the three base LLMs used to show that GoalAct's gains transfer across model families.","marker":"[22]"}],"fun_headline_variants":["GoalAct: global planning and hierarchical execution lift LLM agents 12.22%","Global plans, hierarchical skills: GoalAct beats baselines on legal tasks","12.22% success boost: GoalAct's plan-then-execute breaks agent dead ends","LLM agents stay on track with GoalAct's global plan and skill breakdown","GoalAct: regenerate global plans, act via skills — wins legal tool tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LegalAgentBench's success rate—the fraction of key answer strings that appear in the model's output—measures real task completion, and that the benchmark has not been shaped by GoalAct's own Search/Coding/Writing skill definitions; if key-answer string matching can be satisfied by phrase-spotting, or if the tasks align with GoalAct's prompt-defined skills, the reported 12.22% improvement may not transfer to other benchmarks or real legal work.","fun_headline_variants_meta":{"raw":{"variants":["GoalAct: global planning and hierarchical execution lift LLM agents 12.22%","Global plans, hierarchical skills: GoalAct beats baselines on legal tasks","12.22% success boost: GoalAct's plan-then-execute breaks agent dead ends","LLM agents stay on track with GoalAct's global plan and skill breakdown","GoalAct: regenerate global plans, act via skills — wins legal tool tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2297,"prompt_tokens":1064,"completion_tokens":1233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":1127}},"tokens_in":680,"tokens_out":1233,"duration_ms":11447,"temperature":1.0,"reasoning_tokens":1127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:59:47.499808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GoalAct and the strongest baseline on a fresh set of legal tasks authored independently of the benchmark team, scoring by exact answer matching and by whether the final answer would satisfy a legal expert, and check whether the 12.22% average improvement persists; if the margin shrinks to zero or reverses, the central claim fails.","supporting_citations":[{"cited_title":"In: International Conference on Applied Engineering and Natural Sciences, vol","cited_arxiv_id":null,"evidence_quote":"Provides the Plan-and-Execute baseline that dynamically updates plans but lacks concrete executable actions."}],"review_version":1}