{"id":"e298c7cd-c552-410e-9cc5-285bc55a1204","arxiv_id":"2510.00182","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-based planners inside an LLM-Modulo TAMP loop have lower success rates and higher planning times than engineered planners, and fast direct variants outperform slow thinking variants in most settings.","lead":"This paper benchmarks 16 ways of plugging a large language model into task-and-motion planning, replacing the task planner or the geometric sampler with Gemini 2.5 Flash. Across thousands of trials, engineered planners consistently beat every LLM-based variant on success rate and speed, though LLM variants can still solve many problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Geometric-details claim is confounded: INTEGRATED differs from PDDL variants in both prompt geometry and joint PDDL+pose generation; an ablation is needed.","rationale":"The reader's weakest-assumption identification is exactly the right one: the geometric-details claim is load-bearing because it is one of the three headline results stated in the abstract, and the experimental design cannot separate the effect of adding geometry to the prompt from the effect of requiring joint PDDL-plus-pose generation. The main negative result—that engineered PDDLStream planners outperform these LLM-based variants on success rate and planning time—is supported by a paired, statistically analyzed design and is not undermined by this confound. The direct-versus-reasoning comparison is also largely independent of the geometry issue. Therefore the appropriate verdict remains CONDITIONAL: accept only if the authors either add the PDDL-GEO ablation or soften the causal wording. The abstract/full-text evaluation-count discrepancy (13,750 vs. 4,950) is a real reporting error that should be corrected, but it is secondary to the scientific claim. No ad hominem or overstatement is intended; the paper is otherwise careful and provides code and data for independent checking.","tokens_in":13519,"tokens_out":7501,"duration_ms":66371,"concrete_test":"Run a PDDL-GEO ablation in Blocked and Packing k=5: use the identical PDDL-only prompt plus the geometric object descriptions from Section V.B.3, but ask for PDDL actions only (no pose outputs). Compare its PDDL-failure counts, success rates, and planning times against PDDL-only and INTEGRATED under the same 50-seed paired protocol with the McNemar/Holm analysis. If PDDL-GEO does not show significantly more PDDL failures than PDDL-only, the geometric-distraction explanation is unsupported and the causal wording must be retracted; if it does, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the causal claim in the abstract and Section VII that 'providing geometric details increases the number of task-planning errors.' The supporting comparison contrasts INTEGRATED with PDDL/PDDL+POSES variants. Per Section V.B.3, INTEGRATED prompts add geometric object descriptions and require the LLM to produce PDDL actions and geometric samples in a single joint generation. PDDL-only and PDDL+POSES variants use a PDDL prompt that deliberately excludes geometric details (Section V.B.1). Thus the two conditions differ on two axes: geometric information and joint-generation burden. The extra PDDL errors in INTEGRATED runs could be caused by asking the LLM to produce two output types at once, or by longer prompts, rather than specifically by geometric distraction. Section VII's claim that the comparison 'reveals' a geometric cause overreads the design; the qualitative log analysis does not separate the axes. This does not threaten the main negative result—engineered TAMP beats these LLM variants on success rate and time—but it is one of the three headline findings and should be either re-analyzed with a proper ablation or softened in wording. Secondary: the abstract reports 13,750 evaluations while the full text describes 4,950 problems; the correct scope should be stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a systematic empirical comparison of 16 LLM-Modulo planners for PDDLStream task and motion planning against two engineered TAMP baselines, ADAPTIVE and BILEVEL, across seven instantiations of three benchmark domains. The LLM variants use Gemini 2.5 Flash in zero-shot mode to generate PDDL plans, pose samples, or both, and the TAMP verifier reprompts on failure. Each condition is run 50 times, and significance is assessed with paired exact McNemar tests and Holm-corrected compact letter displays. The central finding is that the LLM-based planners generally have lower success rates and higher planning times than the engineered baselines; direct (non-thinking) variants outperform thinking variants in most comparisons; and INTEGRATED variants, which jointly generate plans and poses, perform worst. The paper interprets these results as evidence that geometric details in the prompt increase task-planning errors and that formal TAMP verification should carry the geometric reasoning burden.","tokens_in":13817,"tokens_out":7087,"duration_ms":65055,"significance":"If the result holds, this is a useful systematic negative result for the LLM-Modulo approach to TAMP. The study is not a single anecdote: it uses a large paired design, exact statistical tests, multiple domains, and public code, prompts, and dialogs. The main success-rate and timing comparisons are well supported and will be of interest to the robotics planning community. The paper is not circular: the baselines are engineered PDDLStream systems from prior work, and no fitted parameters or author-specific assumptions enter the evaluation. However, one of the three headline findings—that geometric details cause more task-planning errors—is not identified by the experimental design, and the abstract contains an inconsistent evaluation count. These issues must be addressed before the paper is published.","major_comments":[{"comment":"The causal claim that 'providing geometric details increases the number of task-planning errors' is not isolated by the comparison. INTEGRATED differs from PDDL-only and PDDL+POSES on two axes: it adds geometric object descriptions to the prompt, and it requires the LLM to produce a PDDL plan and pose samples in a single joint generation. The extra PDDL errors in Fig. 7a could therefore come from the joint-generation burden, longer prompts, or output-format demands rather than from geometric information per se. The sentence in Section VII stating that these methods 'only differ in the PDDL prompt in the inclusion of geometric information' is inaccurate. Since this is one of the three headline findings, please add an ablation (e.g., INTEGRATED without geometric descriptions, or PDDL generation with geometric descriptions appended) or rephrase the claim as an association rather than a caus","section":"Section V.B.3 and Section VII, Fig. 7a"},{"comment":"The abstract reports '13750 evaluations,' but the full text reports 4,950 problems (Section I) and a protocol of 50 trials per algorithm-domain combination (Section VI-B). The described design yields 4,950 runs: 18 conditions in Blocked and the three Packing domains and 9 conditions in the three Rovers domains, each with 50 trials. Please state one consistent count and define what counts as an evaluation. A mismatch of this magnitude in the headline quantitative claim undermines reader confidence in the reported scope.","section":"Abstract vs. Sections I and VI"}],"minor_comments":[{"comment":"The sentence 'our early tests suggested that including geometric details instead harms performance' is presented without supporting data. Since this motivated the design choice for all PDDL prompts, please include the early-test results in an appendix or clearly label this as a preliminary observation.","section":"Section V.B.1"},{"comment":"The Wilcoxon timing comparisons are performed only on successful runs and only for algorithms with success rate at least 30%. Please report the exact number of pairwise comparisons included, and note the selection bias that arises when comparing only on problems where both algorithms succeed.","section":"Section VI-C, Fig. 6"},{"comment":"The footnote states that the Gemini API does not guarantee deterministic outputs even when the seed is set. Please also report the model version/date, sampling temperature, and any other decoding parameters used, so that the experiments are as reproducible as the API permits.","section":"Section VI-B, footnote 2"},{"comment":"The text says 'all LLM-based methods solve a small portion of the problems in Rovers domains' and later 'success rates below 15%.' Please add the exact baseline success rate for ADAPTIVE in Rovers for comparison, since the CLD in Fig. 4(e)–(g) already shows it as the only high-performing method.","section":"Section VII"}],"recommendation":"major_revision","confidential_remarks":"The central negative result—that engineered TAMP beats these LLM-Modulo variants on success rate and time—is credible and well supported. The main obstacle to acceptance is the causal interpretation of the geometric-details result; if the authors run the requested ablation or soften the claim, the remaining issues are largely presentation-level. The evaluation-count inconsistency in the abstract should be corrected before the paper is made public. I do not see evidence of circularity or methodological fabrication; the paired design and release of artifacts are strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is a genuinely useful systematic comparison that gives practitioners a clear negative result—wrapping Gemini 2.5 Flash in an LLM-Modulo loop over PDDLStream loses to engineered TAMP on success rate and time—but one of its three headline claims, that geometric details cause more task-planning errors, is built on a comparison that changes two things at once. The direct-over-thinking finding and the main negative result are well supported.\n\nWhat's new: instead of one-off demonstrations, the paper defines a design space (base algorithm × LLM role × thinking budget) and tests 16 variants on three existing PDDLStream domains with 50 paired trials per condition, exact McNemar tests with Holm correction, and a code/data release. That is a real contribution. The finding that direct generation beats thinking variants in most cases is robust and has practical value: the TAMP system can fix bad plans, so spending compute on the LLM's hidden reasoning is wasteful. The failure-mode analysis (timeouts vs giving up vs token limits) is also useful.\n\nThe soft spots: the claim that geometry hurts task planning is confounded. INTEGRATED prompts add geometric descriptions but also require the LLM to produce PDDL actions and samples in a single call; the PDDL-only and PDDL+POSES variants don't. So the extra PDDL errors in INTEGRATED could come from joint-generation burden or prompt length, not from geometric distraction. The paper's log analysis doesn't separate those axes. This doesn't damage the main result—engineered TAMP wins regardless—but it's one of the three advertised findings, so it needs an ablation (e.g., include geometry in a PDDL-only prompt without joint generation) or softer wording. Secondary: the abstract reports 13,750 evaluations while the full text says 4,950 problems; the discrepancy should be fixed.\n\nThere's a minor point: the study uses only one LLM (Gemini 2.5 Flash) and zero-shot, so the scope is narrower than the title's \"Systematic Study\" might imply. The authors acknowledge this as a limitation, so it's a scope caveat, not an error.\n\nBottom line: this is a solid empirical paper that deserves serious refereeing. The statistics are appropriate, the baselines are standard, and the code and data are out there for checking. The geometric-details claim needs reanalysis or softening, and the evaluation count needs reconciling. For a reader in LLM planning or TAMP, it's worth reading; I'd cite it.\n\nRecommendation for peer review: send it out. With the confound addressed (or softened), it's an accept; as-is, it's a conditional.","headline":"Useful systematic negative result on LLM-Modulo TAMP, but the geometric-details headline is overclaimed due to a confounded comparison.","tokens_in":14274,"tokens_out":2692,"would_cite":true,"duration_ms":21961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot LLM planners inside a formal task-and-motion-planning loop are less successful and slower than the engineered planners they replace; geometric detail in prompts makes task errors worse.","keywords":["large language models","task and motion planning","PDDLStream","LLM-Modulo","zero-shot planning","geometric reasoning","planner evaluation"],"falsifier":"Rerun the INTEGRATED planner with the geometric object descriptions stripped from the prompt while keeping the joint-generation requirement; if the PDDL error rate drops to the level of PDDL-only variants, the added errors are caused by geometric detail, and if it stays high, the cause is joint generation. Either outcome settles the paper's causal claim.","tokens_in":13417,"feed_emoji":"🤖","tokens_out":6000,"duration_ms":43804,"temperature":0.7,"pith_summary":"This paper asks whether an LLM-Modulo framework—where a large language model proposes plans and an engineered verifier checks and corrects them—is a viable strategy for task and motion planning (TAMP). To answer, it constructs 16 planners by replacing parts of two existing PDDLStream-based TAMP solvers with calls to a single LLM, varying what the LLM produces (action plans, geometric poses, both separately, or both in one integrated response) and whether the LLM is allowed to 'think' before answering. Across 4,950 zero-shot problems in three domains, every LLM variant had lower success rates and higher planning times than the engineered baselines. The paper also reports that providing geometric scene details increased the number of task-level (PDDL) errors, and that fast direct generation beat slower reasoning variants in most comparisons. A sympathetic reader would take away that, for now, the efficient division of labor is quick LLM proposal followed by formal TAMP verification.","feed_headline":"Zero-shot LLM planners lose to engineered TAMP planners","feed_subtitle":"A 16-variant head-to-head across 4,950 robot problems finds engineered planners faster and more reliable.","key_machinery":"The carrying mechanism is the LLM-Modulo loop built on PDDLStream: the LLM generates candidate symbolic plans, sampled poses, or both; the base TAMP algorithm (ADAPTIVE or BILEVEL) verifies candidate outputs against stream certificates and PDDL preconditions, reprompting the LLM on failure. The 16 algorithms arise from crossing base algorithm (ADAPTIVE/BILEVEL), the component the LLM substitutes (PDDL planning, pose-stream evaluation, both separately, or integrated generation), and thinking budget (DIRECT vs THINKING). PDDLStream's 'streams'—procedures that certify geometric facts like stable placement or collision-free motion—are what let the verifier do formal geometric checking that the L","core_discovery":"The paper's central claim is that LLM-based planners embedded in an LLM-Modulo TAMP loop do not match engineered TAMP systems, and that design choices matter less than the presence of the verifier. In its zero-shot experiments, the LLM solved many problems but succeeded less often and far more slowly than the base solvers; most LLM failures were timeouts, with a substantial minority being the LLM 'giving up' on solvable problems. The authors further claim that augmenting prompts with geometric information increases PDDL-level errors for non-thinking variants, contrary to the intuition that geometry-aware prompting helps integrated reasoning, and that DIRECT (zero thinking budget) variants ou","pith_inferences":["A testable extension: rerun the integrated variants with geometric descriptions removed from the prompt but with the same joint-generation requirement; if PDDL errors remain high, the true driver is dual-task generation, not geometric distraction.","The finding that verification beats internal thinking suggests a general design principle for LLM-Modulo systems beyond TAMP: keep proposal generation cheap and put all expensive reasoning in the formal checker.","Because the study uses a single LLM and three domains, the headline result may understate or overstate what future models can do; evaluating the same 16 architectures on a stronger model would clarify whether the bottleneck is this LLM or the modular design.","The 'giving up' failures suggest an easy intervention: instruct the LLM to output partial plans or pose candidates even when it doubts global solvability, letting the verifier judge."],"forward_implications":["Engineered TAMP solvers remain more reliable and faster than zero-shot LLM-based planners on these problems.","In an LLM-Modulo loop, the efficient configuration is fast direct LLM generation followed by formal verification; spending LLM computation on internal reasoning is usually not repaid.","Adding geometric detail to prompts does not help the LLM plan better and can degrade its symbolic planning, at least for non-thinking variants.","LLMs can solve some novel TAMP problems zero-shot when embedded in a verifier loop, but resource limits (timeouts, token caps) account for most failures.","Failure analysis shows LLMs sometimes assert that solvable problems are unsolvable, a failure mode that extra computation will not fix."],"fun_headline_variants":["LLM planners trail engineered TAMP in speed and success","Engineered TAMP beats LLM planning across 13,750 tests","LLMs fail to match engineered planners in TAMP tasks","Direct LLM variants outpace reasoning ones in planning","Geometric prompts worsen LLM planning errors"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim that geometric details increase PDDL errors assumes the integrated variant differs from the PDDL-only variant only by the added geometric information, but integrated variants also require the LLM to produce actions and samples in a single joint response, so the extra errors could come from the joint-generation burden instead.","fun_headline_variants_meta":{"raw":{"variants":["LLM planners trail engineered TAMP in speed and success","Engineered TAMP beats LLM planning across 13,750 tests","LLMs fail to match engineered planners in TAMP tasks","Direct LLM variants outpace reasoning ones in planning","Geometric prompts worsen LLM planning errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1629,"prompt_tokens":685,"completion_tokens":944,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":864}},"tokens_in":429,"tokens_out":944,"duration_ms":7660,"temperature":1.0,"reasoning_tokens":864,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T13:26:51.594234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the INTEGRATED planner with the geometric object descriptions stripped from the prompt while keeping the joint-generation requirement; if the PDDL error rate drops to the level of PDDL-only variants, the added errors are caused by geometric detail, and if it stays high, the cause is joint generation. Either outcome settles the paper's causal claim.","supporting_citations":[],"review_version":1}