{"id":"61badcfd-806f-4c6f-af82-a493d6a8ea3b","arxiv_id":"2512.11736","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Bench-Push provides four standardized pushing-based navigation and manipulation environments, new efficiency/effort metrics, baseline policies, and physical-robot validation.","lead":"This paper introduces Bench-Push, an open-source benchmark for testing mobile robots that must push or nudge objects out of the way instead of only avoiding them. It provides four simulated tasks, new scoring metrics for efficiency and interaction effort, and baseline policies that transfer to a physical robot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"E_manip's MST-based L* is not a true lower bound; metric can exceed 1 and is layout-dependent, undermining a central claimed contribution.","rationale":"The reader identified E_manip's dependence on a non-lower-bound MST as the weakest assumption. This is indeed the most load-bearing concern because it directly affects a central claimed contribution—the novel metrics—and the paper's own text admits the normalization is not guaranteed. I considered the other issues (3 physical episodes, anonymous ice-field source, oracle perception in sim-to-real) but they are limitations that can be addressed by softening claims or adding details, whereas the metric issue undermines the quantitative evaluation framework itself. The concrete test would settle whether the MST can overestimate the true optimal path; if it does, the metric's scale is broken and values are not comparable. The reader's CONDITIONAL verdict is appropriate, so no change is needed.","tokens_in":12229,"tokens_out":7526,"duration_ms":64622,"concrete_test":"Construct a Box-Delivery instance with three objects placed collinearly between the robot and the receptacle, such that pushing the nearest object toward the receptacle also delivers the other two. Compute the true minimal robot path length (e.g., by enumerating all feasible pushing sequences) and the MST-based L* defined in Sec. III-C.2. If L*_MST > L*_true, then E_manip > 1 for a robot that pushes straight to the receptacle, proving the metric violates its stated normalization and is not a valid lower-bound efficiency score.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. III-C.2, E_manip is defined as L*(O')/l0, where L*(O') is computed as the weight of an MST of a graph whose vertices are the robot, object starts, and each object's closest goal point. Edges are restricted to object-object, robot-object, and object-own-goal pairs. Because the graph omits edges between goals and other objects/goals, the MST is not a lower bound on the robot path length. If a single pushing action moves multiple objects to their goals (e.g., boxes aligned along the robot's path to the receptacle), the true optimal path can be shorter than the MST weight. Consequently, E_manip can exceed 1, and the overestimation bias varies with object layout, making E_manip values incomparable across episodes. The paper acknowledges the [0,1] guarantee is lacking, but still presents E_manip as a core efficiency metric; without a consistent reference, Eq. (4) does not measure what it claims. Since 'novel evaluation metrics' is a headline contribution, this threatens the benchmark's evaluation validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Bench-Push, a benchmark for pushing-based mobile robot navigation and manipulation. It provides four configurable environments in both 2D (Pymunk) and 3D (MuJoCo) simulation—Maze, Ship-Ice, Box-Delivery, and Area-Clearing—together with a set of novel evaluation metrics (E_nav, I_nav, S_manip, E_manip, I_manip). The benchmark includes reference baselines (SAC, PPO, RRT, SAM, ASV planners) and reports evaluations in simulation and on a physical TurtleBot testbed, claiming zero-shot sim-to-real transfer. The central claim is that this is the first unified benchmark for pushing-based mobile robot navigation and manipulation tasks.","tokens_in":12433,"tokens_out":10456,"duration_ms":93791,"significance":"If the metrics are sound, Bench-Push fills a recognized gap: the field of pushing-based robotics lacks standardized environments and evaluation protocols. The 2D/3D dual-simulation design, configurable environment variations, and bundled baseline implementations are practical contributions that could improve reproducibility and cross-comparison. The open-source Python library and the physical testbed demonstration are useful resources. The paper is transparent about limitations, and the environment descriptions are detailed enough for reimplementation. However, the significance of the contribution depends on the validity of the proposed metrics, particularly E_manip, whose reference path is computed with a heuristic that the authors admit is not a strict lower bound. The sim-to-real claim is also supported by limited data. These issues need to be addressed before the benchmark can be relied upon as a standard evaluation tool.","major_comments":[{"comment":"The manipulation efficiency metric E_manip is defined as L*(O')/l0, where L*(O') is computed as the weight of an MST on a graph whose edges are restricted to object-object, robot-object, and object-own-goal pairs. As the authors state, this MST is not a strict lower bound on the true optimal path length, so E_manip can exceed 1 and its bias can vary with the object layout. This undermines the metric's comparability across episodes and its interpretation as an 'efficiency' score. Since novel metrics are a headline contribution, this is load-bearing. The authors should either replace L* with a true lower bound (e.g., solve an appropriate TSP/Steiner-tree problem on a graph with all relevant transitions) or redefine E_manip as a structural score with an established range, validating its behavior across random layouts. At minimum, an empirical study of E_manip's distribution over the benchma","section":"Sec. III-C.2, Eq. (4)"},{"comment":"The claim that 'policies trained in Bench-Push transfer well to real robots' is supported by only 3 physical episodes per condition, with no statistical tests. The reported rankings are not always consistent between simulation and testbed: in Table III, 10-Obs, RRT has the highest E_nav in simulation (0.758) but the lowest in testbed (0.681), whereas PPO is second in simulation and first in testbed. In Table IV, 5-Obs, SAM's E_manip is higher in the testbed (0.417) than in simulation (0.243), which is not a clear 'alignment'. The conclusion of consistent sim-to-real behavior is stronger than the evidence. Please temper the claim to a feasibility demonstration, or add more trials and appropriate statistical analysis (e.g., hypothesis tests or confidence intervals on rankings).","section":"Sec. IV-C/IV-D, Tables III-IV"}],"minor_comments":[{"comment":"The phrase '(a lower bound on)' in the definition of L*(O') is misleading given the subsequent admission that the MST is not a strict lower bound. Clarify the wording.","section":"Sec. III-C.2"},{"comment":"The physical testbed uses an overhead camera, but the policies expect egocentric observations (occupancy, distance transforms, heading encoding). The paper does not describe how the overhead detections are converted into these observations. This is necessary for reproducibility of the sim-to-real experiments.","section":"Sec. IV-B"},{"comment":"The labels '3-Obs', '6-Obs', '10-Obs' are not defined in the captions or text. State explicitly that these denote the number of movable obstacles/boxes.","section":"Tables III and IV"},{"comment":"The denominator in Eq. (2) uses 'i=0' to include the robot, but the preceding text defines m_i and l_i only for objects i=1,...,K. Introduce m_0 and l_0 explicitly before the equation.","section":"Sec. III-C.1, Eq. (2)"},{"comment":"The abstract says the library 'will be open-sourced', while the body (Sec. I) says 'will also be made available' and the repository URL is given. Use consistent tense and clarify the current availability status of the code.","section":"Sec. I"},{"comment":"There are minor formatting inconsistencies in the bibliography, e.g., 'V .' instead of 'V.', and some entries lack page ranges or venue formatting. A thorough proofread of the references is advisable.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is the E_manip metric, which is a core contribution but rests on an approximation that the authors acknowledge is not a lower bound. The paper is transparent about this, but for a benchmark intended to standardize evaluation, a metric that can exceed 1 and whose bias is layout-dependent is problematic. I believe this is fixable by replacing the MST-based L* with a properly computed lower bound (or by redefining and validating the metric), so I do not recommend rejection. The sim-to-real claim is also thin, but that is secondary. The benchmark infrastructure, environment design, and other metrics are solid, and the code release will be valuable to the community. The 'first unified benchmark' claim is difficult to verify from the manuscript alone but is not a blocker. I would be willing to accept after a revision that addresses the E_manip issue and tempers/supports the sim-to-real claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bench-Push is a real contribution. The subfield of pushing-based robot navigation and manipulation has needed a common evaluation suite, and this paper delivers one: four environments (Maze, Ship-Ice, Box-Delivery, Area-Clearing), each in 2D and 3D, Gymnasium integration, RL and task-specific baselines, and initial physical robot experiments. The navigation metrics E_nav and I_nav are clean and interpretable, and the manipulation metrics, especially S_manip for partial completion, fill a genuine gap. The environment descriptions are detailed enough to reimplement, and the physical experiments, while small, show sensible sim-to-real trends.\n\nThe main soft spot is E_manip. The MST-based L*(O') is explicitly not a strict lower bound, and the authors admit E_manip can exceed 1. That undercuts one of the headline 'novel metrics' — the efficiency score is not benchmarked against a well-defined optimum. This is not fatal, because the paper is transparent about it and doubles of the MST still give a practical path, but it does mean E_manip's values are layout-dependent and not comparable across episodes in a principled way. The fix is either a true lower bound (hard) or a relabeling as a heuristic efficiency proxy rather than a lower-bound-based score.\n\nOther soft spots are minor. The sim-to-real claim rests on three episodes per configuration, with no statistics; that is enough to show trends, not transfer guarantees. The Ship-Ice ice fields come from an anonymous source, which blocks verification. And the 'zero-shot' phrasing is generous given the oracle perception setup with overhead tracking. None of these undermine the central benchmark contribution.\n\nI disagree with any reviewer who would desk-reject this. The benchmark fills a real need, the code is promised open-source, and the authors are honest about limitations. A serious referee could push on the metric definitions and the strength of the sim-to-real claims, but the core infrastructure and the navigation metrics are solid. I would bring this to a reading group focused on robot evaluation methodology, and I would cite it if I were working on pushing-based tasks.\n\nRecommendation: send to peer review. It needs a round of revision, not rejection.","headline":"A genuinely useful, well-scoped benchmark for pushing-based navigation and manipulation, with one headline metric (E_manip) that needs rethinking but not fatal; worth a serious referee.","tokens_in":12949,"tokens_out":1174,"would_cite":true,"duration_ms":12815,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Bench-Push, the first unified benchmark for pushing-based mobile robot navigation and manipulation, with new metrics for efficiency, effort, and partial success.","keywords":["benchmark","pushing-based navigation","pushing-based manipulation","evaluation metrics","sim-to-real transfer","movable obstacles","non-prehensile pushing","partial task completion"],"falsifier":"Build or simulate a box-delivery instance where the true optimal pushing path for the completed sub-tasks is shorter than the MST estimate; if E_manip then exceeds 1 or the estimate fails to lower-bound the actual path, the metric's interpretation breaks.","tokens_in":12041,"feed_emoji":"🤖","tokens_out":4182,"duration_ms":35499,"temperature":0.7,"pith_summary":"The paper argues that pushing-based mobile robot research has been held back by ad hoc environments and inconsistent metrics, and that a shared benchmark can fix that. It introduces Bench-Push, which covers four task families—maze navigation with movable obstacles, ship navigation through ice, box delivery, and area clearing—each in fast 2D and realistic 3D simulation. The benchmark's core contribution is a set of metrics that separate how efficiently a robot reaches a goal or completes a manipulation from how much pushing effort it expends, and that credit partial task completion. The paper also provides reference policies and reports that policies trained in Bench-Push's 3D simulation transfer to a physical robot with consistent ranking of algorithms.","feed_headline":"Bench-Push unifies pushing robots under one scoring system","feed_subtitle":"Five metrics separate path efficiency from interaction effort, and sim policies transfer to a real robot.","key_machinery":"The carrying mechanism is the benchmark's metric design plus its environment abstraction. The navigation metrics use work ratios derived from masses, path lengths, and a constant friction coefficient; the manipulation metrics add a success fraction and a lower-bound path length L*(O') computed from a minimum spanning tree over a graph of object starts, goal contact points, and the robot. This MST estimate is what allows E_manip to approximate 'idealized' efficiency for a partially completed multi-object task, though the paper notes it is not a strict lower bound. Around this, the benchmark provides a standard reinforcement-learning interface, interchangeable bumpers, wheeled and unwheeled bo","core_discovery":"The central claim is that a single benchmark can support both navigation-centric and manipulation-centric pushing tasks while measuring the qualities that matter for pushing: path efficiency, interaction effort, and partial completion. Bench-Push implements this with five metrics: for navigation, E_nav compares the robot's path length to the shortest collision-free path while I_nav is the ratio of work spent moving the robot to work spent moving all objects; for manipulation, S_manip is the fraction of objects successfully delivered or cleared, E_manip compares the executed path to an idealized minimum path, and I_manip is the ratio of necessary work to total work. By providing baselines and","pith_inferences":["A natural extension is to certify the L*(O') lower bound—for small instances one could solve exact shortest-push sequences and place a hard bound under E_manip; if the MST estimate can exceed 1 or fail to lower-bound, the metric would need renormalization.","The work-ratio framing could generalize to other non-prehensile contact tasks, such as sweeping, towing, or excavating, where efficiency of the robot's own motion versus effort applied to the environment is the natural comparison.","The current bird's-eye, known-pushable-object assumption implies the benchmark suits structured lab scenarios; adding egocentric perception and pushability estimation would extend it to partially known or cluttered real-world settings."],"forward_implications":["Researchers can compare pushing policies on identical tasks and score them on efficiency–effort trade-offs instead of success alone.","The 2D simulation tier lets users prototype quickly before committing to expensive 3D or hardware experiments.","Because interaction effort is scored separately from efficiency, policies that nudge obstacles lightly can be distinguished from those that bulldoze through them.","Bench-Push's reported sim-to-real consistency in Maze and Box-Delivery suggests the benchmark can serve as a training-and-evaluation loop for real deployment."],"fun_headline_variants":["One benchmark to score all pushing robots","Pushing robots get a unified yardstick","Bench-Push: standardizing robot pushing trials","Five metrics score pushing robots fairly","Bench-Push measures effort, not just speed"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The manipulation efficiency score E_manip relies on a minimum-spanning-tree estimate of the ideal path length that the paper admits is not a strict lower bound, so if that estimate is not reliably tight, E_manip may no longer mean what it claims.","fun_headline_variants_meta":{"raw":{"variants":["One benchmark to score all pushing robots","Pushing robots get a unified yardstick","Bench-Push: standardizing robot pushing trials","Five metrics score pushing robots fairly","Bench-Push measures effort, not just speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1150,"prompt_tokens":721,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":465,"tokens_out":429,"duration_ms":4379,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T16:46:49.371713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build or simulate a box-delivery instance where the true optimal pushing path for the completed sub-tasks is shorter than the MST estimate; if E_manip then exceeds 1 or the estimate fails to lower-bound the actual path, the metric's interpretation breaks.","supporting_citations":[],"review_version":1}