{"id":"7928c653-7daf-4c75-a6a6-5157933c237c","arxiv_id":"2607.05780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"FORGE decouples robotic tool-use into keypoint trajectory prediction from action-free data and action grounding from limited demonstrations, achieving over 2X improvement in functional generalization to unseen tools.","lead":"This paper introduces FORGE, a two-stage robot policy that uses 2D keypoint trajectories as an intermediate representation to generalize tool-use skills (e.g., hitting) to unseen tools. A smart generalist might read it to understand how decoupling functional reasoning from action execution enables robots to repurpose novel objects for tasks.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Pre-specified hitting points remove the contact-region identification step that the paper itself identifies as central to functional generalization, leaving the 2X improvement attributable to motion conditioning rather than functional reasoning.","rationale":"The reader correctly identifies the 2D representation limitation and the pre-specification of hitting points as concerns, but frames the 2D issue as the load-bearing one. I think this is backwards: the 2D limitation affects a subset of failure cases (the authors note 'imprecise hitting' as one of two failure modes), while the pre-specification of hitting points affects the entire experimental design and the interpretation of what FORGE actually demonstrates. The paper's own framing in Section 1 explicitly lists 'locating its contact region' as a core challenge of functional generalization, yet the experiments bypass this step entirely. The 2X improvement is real and the two-stage decomposition is a reasonable design, but the claim that FORGE achieves 'functional generalization' is overstated relative to what is tested: motion generalization conditioned on pre-identified functional regions. The verdict should remain CONDITIONAL, with the condition being that autonomous contact-region identification is demonstrated. The reader's verdict is directionally correct but the weakest_assumption should be re-prioritized. Additionally, the evaluation scope (single function, 3 unseen tools in main sim experiment) is narrow, though this is secondary to the pre-specification issue. The ATM ablation does provide evidence that function-aware keypoints matter beyond generic tracking, which is a genuine strength, but it does not address whether the functional region identification itself can be learned.","tokens_in":13673,"tokens_out":3555,"duration_ms":220371,"concrete_test":"Run FORGE in a setting where the hitting point is NOT pre-specified. Instead, require the system to autonomously identify the functional contact region on unseen tools—e.g., by training a keypoint proposal model on seen tools and evaluating whether it correctly identifies striking regions on unseen tools, then feeding those into the existing pipeline. If the end-to-end success rate drops below ~0.20 (approaching DP's 0.17), the current 0.36 performance depends critically on external specification of functional regions, and the 'functional reasoning' claim is substantially weakened. If performance remains above ~0.30, the keypoint trajectory prediction genuinely captures transferable functional intent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper frames functional generalization as requiring three capabilities (Section 1): 'locating its contact region, aligning it, and producing an appropriate motion.' However, the experimental setup provides the hitting point and target point as inputs to all methods (Section 4.1: 'the task specifies the hitting point on the tool and the target point on the object'). This means the hardest part of functional reasoning—identifying WHICH part of an unseen tool serves the function—is externally solved. FORGE's Stage 1 (keypoint motion predictor) then operates on keypoints sampled around a pre-specified contact region (Appendix A.1: 'Given the initial hitting point, we use SAM2 to segment the corresponding tool mask and sample N keypoints'). So Stage 1 is performing motion prediction for pre-identified points, not functional understanding. The 2X improvement over baselines (0.36 vs 0.17 for DP) may reflect better motion conditioning via structured keypoint trajectories rather than the transferable functional reasoning the paper claims. The reader's concern about 2D representation insufficiency affects a subset of failure cases; the pre-specification affects the entire experimental design and the interpretation of the central contribution. The authors acknowledge this in limitations but frame it as a minor extension ('future work may leverage vision foundation models to automatically identify function-relevant keypoints'), when it is in fact the core of what 'functional generalization' means.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper proposes FORGE, a two-stage policy for functional generalization in robotic tool use. The key idea is to decouple functional reasoning (predicting 2D keypoint trajectories from action-free data) from grounded execution (mapping keypoints to robot actions with limited demonstrations). The paper introduces a seven-tool hitting-function benchmark and shows that FORGE achieves ~2X improvement in average success rate over baselines (DP, FM, ATM) on unseen tools in both simulation and real-world experiments. Ablations compare intermediate representations (affordance images, human video prompts, keypoint trajectories) and test the effect of action-labeled tool diversity.","tokens_in":13902,"tokens_out":1961,"duration_ms":206476,"significance":"The paper addresses a meaningful problem—generalizing tool-use skills to novel tools that serve the same function. The two-stage decomposition that allows the keypoint motion predictor to train on action-free data while the execution policy trains on a smaller action-labeled set is a reasonable and practical design. The systematic comparison of intermediate representations (Table 2) and the function-aware vs. generic keypoint comparison (ATM baseline) are informative. Real-world validation on multiple unseen tools (Tables 6, Fig. 7) strengthens the empirical contribution.","major_comments":[{"comment":"Section 1 defines functional generalization as requiring three capabilities: 'locating its contact region, aligning it, and producing an appropriate motion.' However, Section 4.1 and Appendix A.1 confirm that the hitting point and target point are pre-specified as inputs to all methods. This means the first capability—identifying which part of an unseen tool serves the function—is externally solved rather than demonstrated by FORGE. The paper's central framing of 'functional generalization' and 'functional reasoning' therefore overstates what the experiments actually test. The 2X improvement (Table 1: 0.36 vs. 0.17 for DP) may be attributable to better motion conditioning via structured keypoint trajectories rather than the transferable functional reasoning claimed. The authors acknowledge this in Section 5 ('the functional-relevant hitting and target points are specified in this study')","section":null},{"comment":"The evaluation is limited to a single function (hitting). While the paper frames functional generalization broadly ('repurposing a book, a stone, or a shoe to drive a nail'), the experimental validation covers only one functional category. This limits the strength of claims about 'functional generalization' as a general capability. The paper should either temper its claims to reflect that this is a single-function study or provide evidence that the approach transfers across functions.","section":null},{"comment":"Table 1: The success metric depends on whether the robot 'strikes the target cube using the designated hitting point' (Section 4.1), but the tolerance for what counts as a successful hit is not specified in the main text. The reader's report flags this as a concern. The paper should state the tolerance explicitly and, ideally, report sensitivity to it, since the absolute success rates (0.36 overall) leave little margin before tolerance choices affect the qualitative conclusions.","section":null}],"minor_comments":[{"comment":"Section 3.2: The representation selection experiment (Fig. 2a) is described as a 'heuristic experiment' but the methodology—how many samples, what success criterion in pixel space, what tolerance—is not detailed in the main text. Table 2 provides results but the protocol should be clarified.","section":null},{"comment":"Table 3: The 1-to-6 setting trains on only the hammer and tests on 6 unseen tools. The 0.21 average SR is reported, but it would be useful to know whether any individual unseen tool performs notably better or worse, as this could reveal tool-specific biases.","section":null},{"comment":"Section 3.3, Eq. (2) and Eq. (5): The flow-matching formulation is standard, but the dimensionality of the keypoint space (N=5 keypoints × 2 coordinates × H horizon) and action space should be stated for reproducibility.","section":null},{"comment":"Appendix A.1: The real-world benchmark is described as containing 6 tools (Table 4) but the main text (Section 4.1) mentions 3 tools for the primary real-world setting and 3 unseen tools in Appendix B.1. This discrepancy should be clarified.","section":null},{"comment":"Figure 2: The notation π_sys1 and π_sys2 is used before being defined. The mapping between 'System-1'/'System-2' naming and 'execution policy'/'keypoint motion predictor' should be made explicit earlier.","section":null},{"comment":"The paper would benefit from reporting standard deviations or confidence intervals on the success rates in Tables 1 and 6, given the moderate sample sizes (30 rollouts per setting in simulation, 10 per pattern in real-world).","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about pre-specified hitting points is the most substantive issue. The paper's own framing in Section 1 explicitly lists 'locating its contact region' as a core capability of functional generalization, yet the experiments provide this as input. This creates a gap between the conceptual contribution (functional generalization) and what is actually demonstrated (motion generalization given a specified contact region). The technical pipeline is sound and the empirical results are informative, but the paper needs to either (a) reframe its claims to accurately reflect what is tested, or (b) add experiments where the contact region is not pre-specified. Option (a) is feasible within revision scope; option (b) may not be. I lean toward major revision rather than reject because the two-stage decomposition and representation comparison are genuine contributions even under the narrower framing."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises three major points: (1) that pre-specifying hitting and target points means FORGE does not fully demonstrate the 'locating' capability claimed in our framing of functional generalization; (2) that evaluation is limited to a single function (hitting), which limits the breadth of claims about 'functional generalization'; and (3) that the success tolerance for hitting is not specified in the main text and sensitivity analysis is lacking. We address each point below and commit to revisions in the revised manuscript.","responses":[{"response":"We agree with the referee that the current experiments do not test the full three-capability definition of functional generalization as stated in Section 1. Specifically, the hitting point and target point are provided as inputs to all methods (including baselines), so the 'locating' capability is not evaluated by our benchmark. We acknowledge that our framing overstates what the experiments demonstrate. In the revised manuscript, we will: (1) revise the definition in Section 1 to distinguish between the full functional generalization problem and the specific sub-problem we evaluate, which we will describe as 'functional motion generalization given specified contact regions'; (2) adjust the abstract and introduction to make clear that FORGE addresses the alignment and motion production components, while contact-region localization is left to future work (as already noted in Section 5); and (3) add a clarifying note that the 2X improvement should be interpreted as evidence that structured keypoint trajectories provide a better intermediate representation for motion conditioning across unseen tools, rather than as evidence that FORGE solves the full functional reasoning pipeline including contact identification. We believe the contribution remains meaningful: the keypoint trajectory representation and two-stage decomposition demonstrably improve cross-tool motion transfer even when contact points are given, and the systematic comparison of intermediate representations (Table 2) provides actionable insights for the broader problem. However, we accept that the claims must be scoped more precisely.","revision_made":"yes","referee_comment":"Section 1 defines functional generalization as requiring three capabilities: locating the contact region, aligning it, and producing appropriate motion. But Section 4.1 and Appendix A.1 confirm hitting and target points are pre-specified, so the first capability is externally solved. The framing of 'functional reasoning' overstates what the experiments test, and the 2X improvement may be attributable to better motion conditioning rather than transferable functional reasoning."},{"response":"The referee is correct that our experimental validation is limited to the hitting function. We chose hitting because it cleanly exhibits the perception-to-action gap—tools with visually different geometries require different motions to achieve the same functional outcome—and because it admits a well-defined success metric. However, we agree that the broad framing of 'functional generalization' in the abstract and introduction is not fully supported by a single-function study. In the revised manuscript, we will: (1) explicitly state in the abstract and introduction that the current work instantiates functional generalization with the hitting function as a first case study; (2) add a sentence in Section 4 noting that extending to additional functions (e.g., cutting, scooping, scraping) is a natural next step and that the keypoint trajectory representation is in principle function-agnostic, but we do not claim empirical evidence of cross-function transfer; and (3) temper claims throughout to use 'functional generalization in the hitting function' or 'single-function functional generalization' where appropriate. We will not add new multi-function experiments in this revision cycle, as we cannot honestly claim results we have not obtained. We note this as a genuine limitation in the revised Discussion section.","revision_made":"yes","referee_comment":"The evaluation is limited to a single function (hitting). While the paper frames functional generalization broadly, experimental validation covers only one functional category. The paper should either temper its claims or provide evidence that the approach transfers across functions."},{"response":"We agree that the success tolerance should be specified in the main text. In our current implementation, a rollout is considered successful if the designated hitting point on the tool makes contact with the target cube, where contact is detected by the simulator's collision checker. The target cube has a side length of 2 cm, so the effective spatial tolerance is approximately 2 cm in the xy-plane. We will add this specification to Section 4.1 in the revised manuscript. Regarding sensitivity analysis: we agree this is a reasonable request and will include a sensitivity analysis varying the tolerance threshold (e.g., at 1 cm, 2 cm, and 3 cm) for the simulation results. We note that the qualitative ranking of methods is unlikely to change substantially, since FORGE's advantage stems from better alignment of the contact region with the target (as shown in the qualitative results), not from marginal tolerance boundary effects. However, we will report the sensitivity data honestly and let the reader assess. If the sensitivity analysis reveals that conclusions are fragile under tighter tolerances, we will report that transparently.","revision_made":"yes","referee_comment":"Table 1: The success metric depends on whether the robot strikes the target cube using the designated hitting point, but the tolerance for what counts as a successful hit is not specified. The paper should state the tolerance explicitly and report sensitivity to it, since absolute success rates (0.36 overall) leave little margin before tolerance choices affect qualitative conclusions."}],"tokens_in":13570,"tokens_out":1160,"duration_ms":161849,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The core thing to know: FORGE is a well-executed two-stage policy that uses flow-matching to predict 2D keypoint trajectories from action-free video, then grounds them into actions with limited demonstrations. The 2X improvement over baselines (0.36 vs 0.17 for DP) is real and the ablations are clean. But the paper frames this as solving 'functional generalization' while externally providing the hitting point and target point to all methods — the hardest part of functional reasoning is bypassed by design, and the gains are attributable to better motion conditioning, not functional understanding.","headline":"Solid empirical recipe for cross-tool transfer, but the 'functional reasoning' claim is overstated — the hardest sub-problem (contact identification) is externally solved.","tokens_in":14437,"tokens_out":198,"would_cite":false,"duration_ms":158852,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Robots learn to swing unfamiliar tools by predicting keypoint motion","keywords":["functional generalization","tool-use manipulation","keypoint trajectories","intermediate representation","robot learning","flow matching","action-free learning","affordance"],"falsifier":"If a 3D keypoint or depth-aware representation replaced the 2D version and substantially improved success rates on the same benchmark, it would suggest that the 2D representation was a bottleneck rather than a sufficient bridge, weakening the claim that 2D keypoint trajectories are the right intermediate.","tokens_in":14000,"feed_emoji":"🔨","tokens_out":1231,"duration_ms":153239,"temperature":0.7,"pith_summary":"This paper tackles a problem the authors call functional generalization: a robot trained to hit a target with a hammer should also succeed when handed a shoe, a book, or a scoop, even though each tool demands a different grip, contact point, and swing path. The core difficulty is that functionally equivalent tools look similar in image space but require entirely different actions, so a standard end-to-end policy that maps pixels directly to motor commands overfits to the tools it was trained on. The authors propose FORGE (Functional Reasoning and Grounded Execution), which splits the problem into two stages. First, a keypoint motion predictor trained on action-free video data forecasts how a handful of tracked 2D points on the tool and target should move over time, capturing the functional intent (where to contact, how to approach) without committing to any particular tool's appearance. Second, a separate execution policy trained on a small set of robot demonstrations translates those predicted keypoint trajectories into actual motor commands. The central claim is that 2D keypoint trajectories are the right intermediate representation for this task, striking a balance between affordance images (too static) and raw video prompts (too entangled with appearance). On a seven-tool hitting benchmark in simulation and real-world experiments, FORGE more than doubles the average success rate of end-to-end baselines on unseen tools, and also outperforms a method that uses generic keypoint tracking rather than function-aware prediction.","feed_headline":"Robots learn to swing unfamiliar tools by predicting keypoint motion","feed_subtitle":"Splitting functional reasoning from action execution more than doubles success rate on unseen tools, with no extra robot data for the hard","key_machinery":"FORGE's pipeline has two stages. Stage 1 trains a conditional flow-matching model (the keypoint motion predictor, called System-2) on action-free data: given a visual observation and current keypoint positions, it predicts future 2D keypoint trajectories. Keypoints are obtained by segmenting the tool mask with SAM2, sampling points via Farthest Point Sampling, and tracking them with CoTracker. Stage 2 freezes the predictor and trains a second flow-matching model (the execution policy, called System-1) on a smaller action-labeled dataset: it takes the predicted keypoint trajectories, adds pixel-level perturbations for robustness, and maps them to robot action chunks. The two-stage factor lets","core_discovery":"The paper's central finding is that decoupling functional reasoning from action execution via 2D keypoint trajectories enables a robot to transfer a hitting function to unseen tools. The keypoint trajectory serves as a bridge representation: it is compact enough to be predicted from action-free observations (no robot data needed for the reasoning stage), structured enough to encode where contact should happen and how the tool should move toward the target, and groundable enough that a small execution policy can convert it into actions. The authors systematically compare this representation against affordance images and human video prompts, finding keypoint trajectories superior on both seen-","pith_inferences":["The two-stage architecture implicitly claims that functional intent is a transferable abstraction separable from motor execution. If this holds, the same decomposition could apply to other perception-to-action gaps in robotics, such as grasping novel objects where visual similarity does not predict grasp kinematics.","The finding that generic keypoint tracking (ATM baseline) underperforms function-aware prediction suggests that not all intermediate representations are equal: the representation must be trained with functional supervision, not just geometric tracking. This raises the question of whether unsupervised discovery of functional keypoints could match the supervised approach.","The benchmark is limited to a single function (hitting) with seven tools. Whether keypoint trajectories remain the best intermediate representation for functions requiring sustained contact, multi-step tool use, or tool-on-tool interaction is untested."],"forward_implications":["If the keypoint-trajectory approach generalizes beyond hitting to other functional tool-use tasks (cutting, scraping, prying), it could reduce the data cost of teaching robots to repurpose novel objects, since the reasoning stage needs no robot demonstrations.","The failure mode of imprecise hitting due to 2D representation limits suggests that extending to 3D keypoint trajectories or depth-aware representations could be a direct next step, which the authors themselves flag.","The monotonic improvement with action-labeled tool diversity (1-to-6: 0.21, 4-to-3: 0.36, 6-to-1: 0.52) quantifies a practical data trade-off: moderate robotic data suffices for grounding, but performance scales with tool coverage, giving practitioners a concrete budgeting curve.","The approach currently requires the hitting and target points to be specified externally; automating function-relevant keypoint identification would be necessary for fully autonomous deployment."],"fun_headline_variants":["Keypoint trajectories let robots generalize hammering to unseen tools","Decoupled reasoning from action doubles robot success on novel tools","Robots transfer hitting function to unfamiliar tools via keypoint motion","2D keypoint trajectories bridge visual function and robot action for tools","Functional reasoning without robot data more than doubles unseen tool success"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire pipeline depends on 2D keypoint trajectories being a sufficient intermediate representation to bridge from perceiving a novel tool to executing the correct action. If the 2D keypoints fail to capture depth or fine-grained spatial alignment needed for precise contact, the downstream execution policy cannot recover, and the authors' own failure analysis confirms this as a real limitation.","fun_headline_variants_meta":{"raw":{"variants":["Keypoint trajectories let robots generalize hammering to unseen tools","Decoupled reasoning from action doubles robot success on novel tools","Robots transfer hitting function to unfamiliar tools via keypoint motion","2D keypoint trajectories bridge visual function and robot action for tools","Functional reasoning without robot data more than doubles unseen tool success"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":623,"prompt_tokens":538,"completion_tokens":85,"prompt_tokens_details":null},"tokens_in":538,"tokens_out":85,"duration_ms":23492,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T00:04:32.231900+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a 3D keypoint or depth-aware representation replaced the 2D version and substantially improved success rates on the same benchmark, it would suggest that the 2D representation was a bottleneck rather than a sufficient bridge, weakening the claim that 2D keypoint trajectories are the right intermediate.","supporting_citations":[],"review_version":1}