{"id":"8a97d72e-2b3a-4a6e-88a3-4575c9f2b01e","arxiv_id":"2506.10897","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GenPlanX couples an LLM-based natural-language to PDDL translation with a classical planner and an execution monitor to solve office automation tasks with plan guarantees.","lead":"GenPlanX is a system that combines a large language model with a classical AI planner so office workers can describe tasks in plain English and have the system generate and execute a step-by-step plan. The paper demonstrates the idea on office tasks like making PowerPoint charts, filtering calendar data, and training a simple prediction model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM-to-dictionary translation is the load-bearing step; Example 1.2 itself shows the generated problem can contradict the user request (db2 cost 3 vs 2), so the planner's guarantee may target the wrong problem.","rationale":"The reader's verdict is CONDITIONAL, and its weakest assumption (LLM translation fidelity) is also the most load-bearing point. All guarantees in the architecture are conditional on the LLM-produced dictionary correctly capturing entities, initial state, and goals; any error makes the classical planner solve a problem that is not the user's. The paper offers no evaluation of this translation, and Section 11's LLM-planning comparison tests direct plan generation, not the NL-to-dictionary step. A concrete red flag appears in Example 1.2: the user request says db2 has reading cost 3, yet the generated initial state and table use cost 2, and the object names 'db2' and 'database2' are mixed. If this is genuine pipeline output, the system would execute a plan optimized against the wrong model; if it is merely a typo, the actual generated dictionary should be shown. Given the absence of reproducible code or a benchmark, the central efficacy claim is not established. The architecture remains plausible, so the correct verdict stays CONDITIONAL; our proposed benchmark would settle whether the translation concern is real.","tokens_in":19383,"tokens_out":7713,"duration_ms":83637,"concrete_test":"Run GenPlanX end-to-end on a benchmark of 50 office requests (including the exact request from Example 1.2), each with a hand-authored gold dictionary and gold optimal plan. Measure exact-match agreement of the LLM-generated dictionary and whether the executed plan achieves the gold goals. If the Example 1.2 run reproduces database2 cost as 2 (not 3) or object-name mismatches, or if fewer than 45 of 50 dictionaries are exactly correct, the planner's guarantee is being applied to the wrong problem and the paper's efficacy claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GenPlanX produces correct, optimal plans because a classical planner generates them after an LLM translates natural language into a structured problem. That guarantee is conditional on the translation being faithful to the user's intent. The paper provides no quantitative evaluation of this translation step, and its own Example 1.2 contains a concrete red flag: the user request states that db2 has reading cost 3, but the LLM-generated initial state in Figure 7 sets '(= (database-cost database2) 2)', and the text and Table 2 use cost 2. Figure 7 also mixes object names 'db2' and 'database2' in the same initial state and omits an opening parenthesis before '(in dataframe1 db2)'. If these reflect actual pipeline output, the planner optimizes a model that does not match the request, and the system would execute the resulting plan with confidence. Since the PDDL correctness guarantee applies only after the translation, this example demonstrates exactly the failure mode the paper must rule out. A single handwritten prompt (Appendix 10.1) with a few-shot examples and a handful of curated successes is insufficient to establish translation reliability across office requests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GenPlanX, a hybrid system for office tasks in which an LLM turns a natural-language request into a Python dictionary containing entities, an initial state, and goals; a compiler converts that dictionary into PDDL; a classical planner (Fast Downward via the Unified Planning library) returns a plan; and an execution module maps PDDL actions to Python functions with monitoring and replanning on failure. The authors describe the architecture, the 'assistant' domain, the full prompt, several worked end-to-end examples (annual-report charts, decision-tree training and prediction, appointments filtering), and a small comparison showing that GPT-4o and o3-mini produce incorrect or suboptimal plans when asked to plan directly. The scientific claims are that this design yields plans that are valid and optimal with respect to the compiled planning problem, and that GenPlanX is effective end-to-end for office-related tasks.","tokens_in":19571,"tokens_out":4638,"duration_ms":50531,"significance":"The architecture is sensible and its division of labor—LLM for translation, classical planner for planning, Python functions for execution—is a credible way to avoid the lack of guarantees in LLM-only planning. The paper is transparent: it provides the full prompt, domain fragments, action implementations, and the raw LLM-planning comparison, which is a strength. If the translation step were shown to be reliable, the system would be a useful contribution in the line of LLM-plus-planner frameworks. However, the central efficacy claim currently rests on a handful of curated examples and no quantitative measurement of the LLM-to-dictionary step. The stress-test concern raised by the reader is real and is borne out by the inconsistencies in Example 1.2. The result is a promising system description rather than a demonstrated end-to-end solution.","major_comments":[{"comment":"The user request states that db2 has reading cost 3, but the LLM-generated initial state in Figure 7 contains '(= (database-cost database2) 2)' and Table 2 and the surrounding text use cost 2 for read-data(db2). Figure 7 also contains a syntactically malformed literal '(in dataframe1 db1) in dataframe1 db2)' and mixes the object names 'db1'/'db2' with 'database1'/'database2'. Because the initial state is the input to the planner, the planner's 'optimal' plan minimizes cost with respect to a problem that does not match the user's stated request. If this is pipeline output, it is a concrete instance of the failure mode the paper must rule out; if it is a typo, it must be corrected and the example re-verified. As written, this section undermines rather than supports the claim that GenPlanX produces the optimal plan for the user's actual request.","section":"§8.1, Example 1.2 / Figure 7 / Table 2"},{"comment":"The paper contains no quantitative evaluation of the LLM's dictionary generation, which is the load-bearing step of the system. There are no reported statistics on translation accuracy, goal completeness, initial-state faithfulness, type correctness, entity-extraction rates, or plan validity of the compiled problems. Section 11 compares LLM-only planning against a classical planner, but that comparison does not test GenPlanX's translation module; if the LLM produces a wrong dictionary, the planner still solves the wrong problem and the error is invisible in that comparison. The single handwritten prompt in Appendix 10.1 and the three curated examples do not establish reliability across the intent list of Section 6.2. A systematic evaluation, for example N requests per intent with human-verified dictionaries and plans, is needed before the end-to-end efficacy claim can be accepted.","section":"§6, §8, §10.1"},{"comment":"The paper claims that GenPlanX monitors execution and replans on failure, but it reports no experiments in which an action fails or a new goal is generated during execution; all worked examples show successful first-pass execution. The success predicates in Listing 5 are essentially type checks, and the text does not specify how a detected failure is translated back into a PDDL state or what guarantees the replanning procedure provides. As written, the execution-monitoring and replanning claim is architectural rather than demonstrated. At minimum, the paper should define the replanning protocol precisely or present a failure-injection study.","section":"§7.2"}],"minor_comments":[{"comment":"The text cites 'Etizioni et al.', but the reference list entry [8] is '0. Etzioni'; the '0.' appears to be a typo for 'O.', and the author name should be harmonized.","section":"§2.1 and References"},{"comment":"References [41] and [42] are identical; one of them should be removed or replaced with the intended distinct source.","section":"References"},{"comment":"The phrase 'problem stated in 2' should read 'problem stated in Listing 2'.","section":"§7.1"},{"comment":"The text says 'shown in Figure 8' when referring to the results table; the correct reference is Table 8.","section":"§11.2"},{"comment":"The sentence 'We also tested with providing the action descriptions in PDDL and got simular results' contains a typo: 'simular' should be 'similar'.","section":"§11.1"},{"comment":"The instruction 'Do not return Output: in the output' is ambiguous; it should be rephrased to make clear whether it means 'do not prefix your response with Output:'.","section":"Appendix 10.1"},{"comment":"Several instances of 'GenPlanX' are missing the space before the word (e.g., 'introducesGenPlanX' in the abstract and 'handle file and data management tasks' formatting issues); these should be fixed in the final formatting pass.","section":"Abstract and headings"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's conditional assessment. The deciding issue is the absence of a quantitative evaluation of the LLM-to-dictionary translation step, combined with the concrete inconsistency in Example 1.2 that demonstrates exactly that failure mode. Because the architecture is sound and the gap is empirical, I see this as fixable in revision rather than grounds for rejection. I would also ask the authors to state whether code and data will be released for reproducibility, since the paper does not currently mention any artifact release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straight to the point: this is a system paper describing GenPlanX, an integration of an LLM that turns natural-language office requests into a structured dictionary, a compiler that turns that dictionary into PDDL, a classical planner, and an execution-and-monitoring layer with replanning. The components are known—the paper itself cites LLM+P, TRIP-PAL, and the LLM-modulo framework—and the architecture is a reasonable instantiation with two small twists: the LLM outputs a dictionary instead of raw PDDL, and the domain is a realistic office assistant with actions for files, databases, presentations, and ML. The paper does a few things well. It gives a complete prompt in the appendix, which is genuinely useful for anyone building a similar copilot. It explains the execution mapping from PDDL actions to Python functions. And it includes a small sanity check showing that GPT-4o and o3-mini alone struggle to produce optimal plans for the same problem, which motivates the hybrid design.\n\nThe soft spot is the evidence. The end-to-end claim—that GenPlanX can reliably translate user intent into a planning problem that matches that intent—is supported only by a handful of curated examples. No translation accuracy, no plan-validity statistics, no success rates, no comparison against an end-to-end baseline. That matters because the planner's guarantee is conditional: it guarantees the plan is valid for the PDDL problem it receives, not for the user's request. If the LLM misstates a cost or a fact, the planner confidently solves the wrong problem.\n\nAnd the paper contains a concrete instance of exactly that failure. In Example 1.2, the user request says db2 has reading cost 3. The LLM-generated initial state in Figure 7 sets (database-cost database2) to 2, and Table 2 and the surrounding text also use 2. Figure 7 also has a malformed literal and mixes db2 with database2. If that output came from the real pipeline, the 'optimal' plan the authors highlight is optimal for a problem the user didn't specify. That is not a minor typo; it's the load-bearing step in action. The stress-test note is correct about this.\n\nThe architecture itself is sound, and the paper is not circular or incoherent. But the central efficacy claim is not supported as written. The fix is straightforward: add a modest evaluation—say, fifty varied requests, report how often the LLM produces the right objects/init/goals, how often the compiled plan is executable, and what happens on failure—and fix the Example 1.2 discrepancy. If the translation reliability isn't there, scale back the claim to 'architecture and examples' rather than 'demonstrate efficacy.'\n\nWho is this for? Researchers or engineers building LLM-plus-planner copilots; they'll get a useful prompt design and a concrete domain. It deserves peer review—the system is real and the translation step is exactly what a reviewer should pressure-test—but I'd expect major revision, not acceptance on the current evidence.","headline":"A clean integration of known LLM-plus-planner components whose load-bearing translation step is unmeasured and, in one example, demonstrably wrong—worth reviewing, not ready to be believed.","tokens_in":20208,"tokens_out":3565,"would_cite":false,"duration_ms":36700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GenPlanX is an architecture that turns natural-language office requests into cost-optimal, executable plans by giving the LLM the job of writing the problem and a classical planner the job of solving it.","keywords":["LLM planning","classical AI planning","PDDL","execution monitoring","replanning","office automation","natural language understanding","softbots"],"falsifier":"Give GenPlanX a request with a known ground truth—'Read annual-report.csv, make a bar chart of balance by year, and email the result to alice@example.com'—and inspect whether the LLM's dictionary contains the email object and the sent-email goal; a well-formed dictionary that omits a stated goal or misrepresents the initial file would lead to an artifact that misses the request while the system reports success. A broader falsifier is a batch evaluation of such requests where the generated problem specification is compared to gold-standard extractions; any miss that the execution loop does not catch would signal that the classical planner's guarantee does not reach the user.","tokens_in":19036,"feed_emoji":"📋","tokens_out":8608,"duration_ms":89690,"temperature":0.7,"pith_summary":"The paper tries to establish that a planning system can take office requests in plain English and carry them out with the formal guarantees classical planning provides, without losing the language understanding only an LLM offers. Its central claim is that GenPlanX—which sends the request through entity extraction, uses an LLM to return a structured problem dictionary, compiles that dictionary into PDDL, solves it with a classical planner, and executes the plan with monitoring and replanning—delivers end-to-end office workflows such as generating a PowerPoint, choosing a cost-minimal database path, and training a decision-tree model. The intended significance is that LLMs alone cannot be trusted for plan correctness or optimality, while classical planners cannot read natural language; the hybrid puts each component where it is strong.","feed_headline":"LLM plus classical planner executes office tasks end-to-end","feed_subtitle":"GenPlanX translates natural language into a formal problem, finds a cost-optimal plan, and replans when execution fails.","key_machinery":"The load-bearing object is the 'assistant' PDDL domain, a curated set of typed objects, predicates, and actions representing office tools and data: types such as data-file, dataframe, graph, slide, presentation, model, and response; predicates such as (in ?c ?c1), (available ?o), (done-query ?q), and (in-graph ?c ?c1 ?g); and actions like read-data, query-data-basic, query-data-optimized, create-graph, add-to-graph, create-slide, generate-presentation, learn-supervised, and predict-using-learned-model. Each action has a Python function of the same name that actually performs the operation on the execution state, so the plan is directly executable. The second load-bearing piece is the LLM-produced dictionary, the only place natural language enters: it specifies the objects, initial state, and goals, and the compiler turns it into PDDL. The separation is what lets the planner's validity guarantee apply to a problem that was authored in English.","core_discovery":"The central discovery is the architecture itself and the division of labor inside it. The LLM never proposes a plan; it only proposes the planning problem, in the form of a Python dictionary whose keys are objects, 'init_state', and 'goals', and whose values carry both PDDL types and execution-time values such as file paths and query strings. A compiler converts that dictionary into a PDDL problem, and a classical planner returns a cost-optimal action sequence. The execution module maps each PDDL action to a same-named Python function, checks after each action whether the real-world effect matched, and invokes replanning on failure or when an action (such as reading an email) reveals new goals. The paper's evidence is three worked end-to-end examples plus a small comparison in which two recent LLMs, when asked to plan directly without hints, produced at most 4/5 correct and 0/5 optimal plans, while both reached 5/5 with hints, supporting the claim that the planner is what supplies correctness.","pith_inferences":["A fair evaluation would separate translation fidelity from planning: measure how often the LLM-produced dictionary matches a validated extraction of objects, initial state, and goals on a held-out set of office requests, because the planner's guarantee cannot compensate for translation error.","Using a Python-dictionary intermediate representation, rather than asking the LLM to write PDDL directly, likely makes the translation step easier for the model and also lets execution values such as file paths and email addresses ride along as object values, a design choice worth adopting in similar systems.","The replanning-on-new-goals behavior invites a goal-driven autonomy extension: the monitoring layer could detect opportunities in the environment and propose new goals, not just wait for an action to fail or explicitly announce them."],"forward_implications":["Plans produced by GenPlanX are provably valid with respect to the problem the LLM generated, so any end-to-end failure can be traced to translation rather than plan search.","Because costs are encoded in domain actions, the planner can optimize over alternatives that LLM-only planning gets wrong, as in Example 1.2 where the lower-cost database route is chosen.","The architecture supports adding new tools by defining a PDDL action and a same-named Python function, making it adaptable to new office applications.","Execution monitoring plus replanning lets the system react to failures and to goals only discovered during execution, such as actions that read email contents and create new intents.","Any PDDL planner can be substituted, making the system planner-independent."],"supporting_citations":[{"why":"Defines the PDDL standard that the compiled problem and domain files are written in, grounding the planner interface.","marker":"[12]"},{"why":"The classical planner used by GenPlanX to compute cost-optimal plans, providing the soundness guarantee.","marker":"[18]"},{"why":"Unified Planning library used to call the planner, supporting planner independence.","marker":"[24]"},{"why":"Supplies the entity extraction module that identifies domain-specific entities for prompt generation, a prerequisite for accurate dictionary output.","marker":"[21]"},{"why":"The LLM+P framework, a prior hybrid that translates natural language to PDDL; GenPlanX differentiates by using a dictionary intermediate and execution monitoring.","marker":"[23]"},{"why":"The LLM-modulo framing that LLMs can help planning but cannot plan alone, motivating the hybrid design.","marker":"[20]"},{"why":"Evidence that LLMs alone struggle to generate valid plans on planning benchmarks, motivating the use of a classical planner.","marker":"[41]"}],"fun_headline_variants":["LLM frames problem, classical planner picks optimal steps","From words to optimal plans: LLM + classical planner","LLM never plans, only frames; planner optimizes","LLM+planner replans when real world diverges","Natural language in, optimal action out: GenPlanX"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM will convert the user's request into a complete and accurate dictionary of objects, initial state, and goals; if it misses a goal or misstates the initial configuration, the planner will efficiently solve the wrong problem and the execution loop will carry out that wrong plan with confidence.","fun_headline_variants_meta":{"raw":{"variants":["LLM frames problem, classical planner picks optimal steps","From words to optimal plans: LLM + classical planner","LLM never plans, only frames; planner optimizes","LLM+planner replans when real world diverges","Natural language in, optimal action out: GenPlanX"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3910,"prompt_tokens":861,"completion_tokens":3049,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":2968}},"tokens_in":477,"tokens_out":3049,"duration_ms":24979,"temperature":1.0,"reasoning_tokens":2968,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:13:01.090576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give GenPlanX a request with a known ground truth—'Read annual-report.csv, make a bar chart of balance by year, and email the result to alice@example.com'—and inspect whether the LLM's dictionary contains the email object and the sent-email goal; a well-formed dictionary that omits a stated goal or misrepresents the initial file would lead to an artifact that misses the request while the system reports success. A broader falsifier is a batch evaluation of such requests where the generated problem specification is compared to gold-standard extractions; any miss that the execution loop does not catch would signal that the classical planner's guarantee does not reach the user.","supporting_citations":[{"cited_title":"Ghallab, A","cited_arxiv_id":null,"evidence_quote":"Defines the PDDL standard that the compiled problem and domain files are written in, grounding the planner interface."},{"cited_title":"The fast downward planning system.Journal of Artificial Intelligence Research, 26:191– 246, 2006","cited_arxiv_id":null,"evidence_quote":"The classical planner used by GenPlanX to compute cost-optimal plans, providing the soundness guarantee."},{"cited_title":"Unified planning: Modeling, manipulating and solving ai planning problems in python.SoftwareX, 29:102012, 2025","cited_arxiv_id":null,"evidence_quote":"Unified Planning library used to call the planner, supporting planner independence."},{"cited_title":"Advanced messaging platform (amp): Pipeline for automated enterprise email processing","cited_arxiv_id":null,"evidence_quote":"Supplies the entity extraction module that identifies domain-specific entities for prompt generation, a prerequisite for accurate dictionary output."}],"review_version":1}