{"id":"b17156d4-2c45-4687-8286-81b013eecc7e","arxiv_id":"2505.03035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MORE uses LLM-based scene graph filtering and instance-level reasoning to solve 48.1% of 81 selected BEHAVIOR-1K rearrangement tasks, outperforming prior LLM and VLM planners in the same modified setup.","lead":"MORE is a robot planning method that filters a large indoor or outdoor scene down to task-relevant objects before asking a language model to choose the next move. On a hand-picked set of 81 BEHAVIOR-1K activities it raised task success from about 20% to 48% under a simulation setup that skips physical arm execution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 48.1% success rate is measured under magic actions and ground-truth semantics, so it supports a planning-only claim, not the claim of solving BEHAVIOR-1K as an embodied mobile manipulation benchmark.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: magic actions and ground-truth semantic perception decouple the evaluation from real mobile manipulation. My stress-test pass confirms this and adds two details: Section S.2 shows that grasp/place are pure precondition checks rather than physical manipulations, and the real-world table contains an inconsistency where the book task is counted complete despite the book falling outside the target container. These do not contradict the paper's internal disclosure—the authors are explicit about the shortcuts—but they do contradict the abstract's and introduction's framing that MORE solves BEHAVIOR-1K tasks as a mobile manipulation method. The relative baseline comparison may still be meaningful for high-level planning, which is why the appropriate verdict remains CONDITIONAL rather than REJECT or UNVERDICTED. The paper should be accepted only with the central claim reframed as planning-level performance, or with additional evidence that the same planner transfers to physically executed manipulation and realistic perception. Since the reader already conditioned the verdict on exactly this issue, no verdict change is needed.","tokens_in":17336,"tokens_out":4956,"duration_ms":58367,"concrete_test":"On a representative subset of the 81 tasks, for example 10 tasks spanning both place-inside and place-ontop goal conditions, run MORE with the planner, scene graph, prompts, and LLM calls unchanged, but replace the magic-action subpolicies with physically simulated Fetch arm motion and grasp/place execution in OmniGibson, and replace ground-truth semantics with an open-vocabulary segmentation model (or a comparable noisy perception pipeline). If the success rate on this physical subset is substantially lower than the same subset under magic actions—for example, more than 10 percentage points lower—then the reported 48.1% SR is a planning-only result and the central claim should be downgraded accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that MORE is \"the first approach to successfully solve a significant share of the BEHAVIOR-1K benchmark\" and that it performs \"mobile manipulation rearrangement.\" But the simulation evaluation explicitly replaces the two hardest parts of that claim with shortcuts. Section III-A states that the method \"assume[s] accurate semantic perception by relying on ground truth semantics,\" and Section S.2 states that perception is abstracted to simulator-provided ground-truth instances and semantic segmentation. More importantly, Section III-A says the accelerated subpolicies follow the concept of \"magic actions,\" removing the need for physical simulation and \"disabl[ing] the precise execution of arm movements.\" Section S.2 confirms that grasp and place actions succeed once the robot has navigated near the object and the relevant preconditions (e.g., arm empty, target container open) are met; no physical grasp, lift, carry, or placement is simulated. Thus the reported 48.1% SR and 80.1% rTP measure high-level action selection on an idealized symbolic state-update abstraction, not whether a real mobile manipulator can complete those rearrangement tasks. The relative ordering of MORE against MoMa-LLM and BUMBLE may survive as a planning comparison, because all methods are evaluated under the same shortcut, but the stronger claim of solving BEHAVIOR-1K in its intended embodied sense is not supported. The real-world section only provides three qualitative tasks and contains a relevant inconsistency: the book task is marked 1/1 completed even though the text says the book \"fell outside box\" before done() was called. The load-bearing issue is therefore not a hidden error in the planner, but a mismatch between the central claim and what the evaluation actually measures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MORE, a language-model-based planner for long-horizon mobile manipulation rearrangement. It constructs a hierarchical 3D scene graph from RGB-D observations, filters the graph to a task-relevant subgraph using an LLM prompt, and uses a second LLM to select object-centric subpolicies in a model-predictive-control loop. The authors add instance differentiation, a Voronoi-graph sparsification scheme, and region separation at doors to support indoor-outdoor scenes. They evaluate MORE on 81 hand-picked tasks from the BEHAVIOR-1K benchmark, reporting a 48.1% success rate and 80.1% relative task progression, outperforming MoMa-LLM and BUMBLE baselines, along with three qualitative real-world tasks. The simulation evaluation relies on ground-truth semantics and 'magic actions' that bypass physical manipulation, as acknowledged in Sections III-A and S.2.","tokens_in":17622,"tokens_out":5774,"duration_ms":56651,"significance":"If the claims are taken at face value, MORE demonstrates that LLM-based planning with explicit scene-graph memory and active filtering can make substantial progress on long-horizon rearrangement tasks at scale. The paper's strengths include a reproducible benchmark subset (public task descriptions and accelerated subpolicies), a clear system description, and a strong improvement over the MoMa-LLM predecessor. However, the reported results are obtained under strongly idealized perception and execution assumptions, so the significance is limited to high-level planning rather than embodied mobile manipulation. The absence of multiple seeds and ablations further tempers the strength of the comparisons.","major_comments":[{"comment":"The evaluation uses ground-truth semantics and 'magic actions' that disable arm movements and bypass physical simulation. Consequently, the reported 48.1% SR and 80.1% rTP measure high-level action selection on a symbolic state-update abstraction, not mobile manipulation in BEHAVIOR-1K as an embodied benchmark. The abstract claims MORE is 'the first approach to successfully solve a significant share of the benchmark' and the title refers to 'Mobile Manipulation Rearrangement'; these claims overstate what is demonstrated. The authors should explicitly qualify all such claims as planning under idealized perception and execution, or add results from actual physical simulation.","section":"Section III-A, Section S.2"},{"comment":"The results are single-run point estimates with no error bars, multiple seeds, or discussion of LLM sampling temperature. Because GPT-4o is stochastic, the 48.1% SR and the relative ordering of methods need statistical support. The metric definition for Success Rate is also self-contradictory: it states SR requires 'done() called correctly' and then says 'this does not require the model call done() correctly after doing so'. Please fix the metric definitions and report variance.","section":"Section IV-A, Table I"},{"comment":"The 81-task subset is hand-selected from the 1,000 BEHAVIOR-1K tasks, with scenes (restaurant diner, school gym) and tasks with failed initialization excluded. The paper should analyze selection bias: how representative are these 81 tasks of the full benchmark, and how many of the 163 tasks involving articulated objects were removed due to initialization failure? Without such analysis, the claim of solving 'a significant share of the benchmark' is only established for a curated subset, and the 'first approach' claim is not meaningful relative to the original benchmark.","section":"Section III-A, Table S.1"},{"comment":"The paper does not ablate MORE's components. The comparison BUMBLE + filtering shows that the filtering alone helps BUMBLE, but no experiment removes filtering, instance differentiation, or Voronoi sparsification from MORE itself. Without ablations such as 'MORE without filtering' or 'MORE without instance IDs', it is unclear which components drive the improvement over MoMa-LLM + spatial relations. Please add at least one such condition to support the claimed contributions.","section":"Section IV-B"}],"minor_comments":[{"comment":"There are typos: 'BUBMLE' should be 'BUMBLE' in the baseline description, and 'halucinations' appears in Section S.5 and should be 'hallucinations'.","section":"Section IV-A, Section S.5"},{"comment":"In the low-level action space, 'turn-left by up to 35 rad' and 'turn-right by up to 35 rad' are presumably meant to be small angles such as 0.35 rad; 35 radians is physically nonsensical.","section":"Section S.2"},{"comment":"The caption text 'BtOccupancy map GV' is garbled; it should read something like 'Occupancy map B_t, Voronoi graph G_V' to clearly label the components.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claim is stronger than the evidence: the evaluation relies on magic actions and ground-truth semantics, and the 81-task subset is author-selected. The method itself is a reasonable extension of MoMa-LLM, and the public release of task descriptions and accelerated subpolicies is useful. I would be comfortable after a major revision that scales back the embodied-manipulation claims, adds multiple seeds and ablations, and discusses the selection bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core take: MORE is a sensible step for LLM-based planning in large scenes, and the 48.1% number is worth taking seriously as a planning benchmark result. The paper is upfront about the conditions: ground truth semantics, magic actions, disabled arm movement, closed exterior doors, and 81 hand-filtered tasks. Under those conditions the comparison against MoMa-LLM and BUMBLE is plausible, and the improvements are meaningful. The scene graph filtering with instance IDs and indoor-outdoor Voronoi sparsification is a real contribution, and the code plus task descriptions are a useful resource for the community.\n\nThe main soft spot is the gap between the claim and the evaluation. The abstract says MORE 'solves' a significant share of BEHAVIOR-1K, and the introduction says it is the first method capable of 'solving' tasks. But the simulation removes physical grasp, lift, and placement; actions succeed as long as the robot is near the object and preconditions hold. That measures high-level action selection on an idealized state abstraction, not embodied mobile manipulation. The relative ordering of methods likely survives because all baselines run under the same magic-action setup, but the absolute success rate should be labeled as planning-only.\n\nOther issues are secondary. The 81-task subset is hand-selected and excludes tasks involving articulated objects and failed initializations; that is disclosed, but it means the number is not comparable to a full-benchmark pass. No error bars or multiple seeds, and the per-task results are averaged across tasks with very different difficulty. The real-world section is qualitative, with three tasks, and the book task is marked 1/1 completed even though the text says the book fell outside the box before done(); the claim that all goal conditions were completed for all tasks is contradicted by their own account. That is a small but real inconsistency.\n\nThe free parameters (lambda, sparsification threshold, door kernel threshold) are implementation choices, not fitted to benchmark outcomes, so no circularity problem. Self-citations are to the predecessor MoMa-LLM, which is appropriate.\n\nWho this is for: researchers working on LLM/VLM planners for long-horizon tasks, scene graph compression, and benchmark subsets for BEHAVIOR-1K. It deserves a serious referee: the method is concrete, the evaluation is broad, and the limitations are mostly disclosed. The authors should be pushed to soften the 'solving' language, add variance estimates or at least per-task breakdowns, and fix the real-world table inconsistency.","headline":"A solid planning-level contribution with honest disclosure of its shortcuts; the 48.1% result should be read as measuring planning under idealized perception and magic actions, not as solving BEHAVIOR-1K end-to-end.","tokens_in":18267,"tokens_out":1655,"would_cite":true,"duration_ms":16228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MORE is the first LLM-based planner to solve a large share of BEHAVIOR-1K rearrangement tasks, reporting 48.1% success across 81 tasks.","keywords":["mobile manipulation","rearrangement tasks","scene graphs","LLM planning","BEHAVIOR-1K","instance differentiation","scene graph filtering","Voronoi graph"],"falsifier":"Run MORE on the same 81 BEHAVIOR-1K tasks with physical Fetch arm control enabled and with a real open-vocabulary segmentation model instead of ground-truth semantics; if the success rate falls far below 48.1% or manipulation steps fail often, the reported result is planning-only. A smaller check is to run 20 trials of each of the three real-world tasks with the N2M2 policy and AR-marker perception, counting completed goal conditions; if completion drops below the reported 3/3, low-level execution is the bottleneck.","tokens_in":17143,"feed_emoji":"🤖","tokens_out":5932,"duration_ms":57640,"temperature":0.7,"pith_summary":"This paper proposes MORE, a planner that uses a large language model to solve long-horizon mobile manipulation rearrangement tasks in large, partially unexplored indoor and outdoor environments. It claims that representing the world as a hierarchical scene graph, filtering that graph down to task-relevant object and region instances, and giving each object instance a distinct identifier lets an LLM plan reliably where prior LLM- and VLM-based planners hallucinate or get lost. On 81 BEHAVIOR-1K tasks, MORE reports a 48.1% success rate and 80.1% relative task progression, the first substantial share of that benchmark solved. The paper also contributes fully specified natural-language task descriptions and accelerated \"magic action\" subpolicies that make large-scale evaluation feasible, alongside real-world demonstrations in a three-room apartment.","feed_headline":"LLM planner solves 48% of 81 BEHAVIOR-1K tasks","feed_subtitle":"Scene-graph filtering and instance IDs shrink the planning problem, beating prior models on long-horizon rearrangement.","key_machinery":"The load-bearing object is the task-informed subgraph $\\xi_{task}$ produced by an LLM-based active filtering function $f: \\xi \\to \\xi_{task}$. The full scene graph, spanning root, regions, and objects with alphabetical instance IDs and inferred attributes, is rendered as structured text; the LLM is prompted to delete objects irrelevant to the task or to finding task-relevant objects, yielding a bounded planning problem that is then given to a second LLM call selecting among object-centric subpolicies (navigate, open, close, grasp, place_inside, place_ontop, explore, done). Supporting machinery includes a Voronoi graph sparsified by removing degree-two nodes with close neighbors in $O(|V|)$ time, region separation at doors using Gaussian kernels matched to door bounding boxes, and assignment of objects to navigable connected components so unreachable regions are excluded from planning.","core_discovery":"The central discovery claimed is that task-relevant subgraph filtering, rather than more powerful reasoning alone, is what unlocks LLM-based planning at scale: by compressing a scene with hundreds of objects into a small subgraph $\\xi_{task}$ before invoking the planner, the LLM's action choices stay grounded. MORE couples this with instance differentiation (alphabetical IDs so the model can track multiple daffodils, boxes, or bottles), a sparsified Voronoi navigation graph, door-aware region separation, and MPC-style replanning after each subpolicy. The result is the first method reported to complete a significant share of BEHAVIOR-1K: 48.1% success rate, 50.6% total task completion, 70.1% task progression, and 80.1% relative task progression across 81 tasks, outperforming MoMa-LLM and BUMBLE with and without the additional filtering.","pith_inferences":["Inference: the active filtering step is a task-relevance bottleneck that could likely be swapped into other LLM- or VLM-based planners; the paper's own BUMBLE-plus-filtering result supports this.","Inference: because arm movements are replaced by magic actions and semantics come from ground truth, the benchmark numbers should be read as planning ability; end-to-end performance in the real world will depend on perception and manipulation components the paper only partially exercises.","Inference: the TP/rTP gap points to an explicit stopping or \"done\" reasoning problem, and future work could add ambiguity resolution by asking the LLM whether unobserved areas might still contain required object instances.","Inference: a testable extension would run the same filtering and planning pipeline with an open-vocabulary perceptual scene graph instead of ground-truth semantics, which would reveal how much of the 48.1% success rate survives realistic perception."],"forward_implications":["MORE reports the first substantial success on BEHAVIOR-1K rearrangement tasks, with 48.1% success and 50.6% total task completion, showing that LLM planners can handle hundreds of objects when given a filtered scene.","Scene graph filtering alone improves a baseline: BUMBLE's success rate rises from 11.3% to 15.7% when filtering is added, indicating the filtering scheme transfers as a module.","Instance differentiation closes part of the gap: MoMa-LLM with spatial relations goes from 19.7% to 36.1%, and MORE reaches 48.1%, so tracking distinct object instances matters for multi-object tasks.","The gap between 70.1% task progression and 80.1% relative task progression suggests most failures come from knowing when to stop searching or satisfying all constraints, not from failing to find the relevant objects.","Real-world demonstrations on set-the-table, clean-up, and fetch-and-store tasks show the same planner can drive an HSR robot in a multi-room apartment, though low-level execution errors such as a late gripper release can still occur."],"supporting_citations":[{"why":"Conceptual predecessor; supplies the Voronoi-based scene graph, structured-language scene representation, and room classification that MORE extends to instance-level reasoning.","marker":"[2]"},{"why":"VLM baseline with visual memory that MORE compares against and also augments with the filtering scheme in an ablation.","marker":"[3]"},{"why":"Source of the 81 BEHAVIOR-1K rearrangement tasks and the benchmark that MORE claims to solve substantially.","marker":"[12]"},{"why":"Defines the BDDL goal conditions from which the paper derives fully specified natural-language task descriptions.","marker":"[39]"},{"why":"Shows that planning over large 3D scene graphs leads to exploding planning times, motivating the filtering step.","marker":"[11]"},{"why":"Exemplifies hallucination when LLMs plan with too many objects; MORE's filtering is designed to avoid that failure mode.","marker":"[33]"}],"fun_headline_variants":["Scene-graph filter tames LLM planning for 48% task success","First LLM planner to crack BEHAVIOR-1K: 48% success","Scene-graph compression makes LLM rearrange real rooms","Filtering scene graphs beats bigger models on 81 tasks","Instance IDs and subgraphs let LLM plan 48% of tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing real arm control with \"magic actions\" and using ground-truth semantic perception still tells us how the method would behave with real perception and manipulation; if those shortcuts do not faithfully represent real-world conditions, the 48.1% success rate supports only the high-level planning component.","fun_headline_variants_meta":{"raw":{"variants":["Scene-graph filter tames LLM planning for 48% task success","First LLM planner to crack BEHAVIOR-1K: 48% success","Scene-graph compression makes LLM rearrange real rooms","Filtering scene graphs beats bigger models on 81 tasks","Instance IDs and subgraphs let LLM plan 48% of tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3744,"prompt_tokens":949,"completion_tokens":2795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2715}},"tokens_in":565,"tokens_out":2795,"duration_ms":20476,"temperature":1.0,"reasoning_tokens":2715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:00:52.933993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MORE on the same 81 BEHAVIOR-1K tasks with physical Fetch arm control enabled and with a real open-vocabulary segmentation model instead of ground-truth semantics; if the success rate falls far below 48.1% or manipulation steps fail often, the reported result is planning-only. A smaller check is to run 20 trials of each of the three real-world tasks with the N2M2 policy and AR-marker perception, counting completed goal conditions; if completion drops below the reported 3/3, low-level execution is the bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Conceptual predecessor; supplies the Voronoi-based scene graph, structured-language scene representation, and room classification that MORE extends to instance-level reasoning."},{"cited_title":"Honerkamp, M","cited_arxiv_id":null,"evidence_quote":"VLM baseline with visual memory that MORE compares against and also augments with the filtering scheme in an ablation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the 81 BEHAVIOR-1K rearrangement tasks and the benchmark that MORE claims to solve substantially."},{"cited_title":"Deitke, E","cited_arxiv_id":null,"evidence_quote":"Defines the BDDL goal conditions from which the paper derives fully specified natural-language task descriptions."},{"cited_title":"Nasiriany, A","cited_arxiv_id":null,"evidence_quote":"Shows that planning over large 3D scene graphs leads to exploding planning times, motivating the filtering step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Exemplifies hallucination when LLMs plan with too many objects; MORE's filtering is designed to avoid that failure mode."}],"review_version":1}