{"id":"734b3513-4158-465d-8949-f4442a728f36","arxiv_id":"2608.08929","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new benchmark and pipeline show that grounding an action word like 'cut' on a familiar object can be done without ever training on that action word, outperforming prior affordance grounding methods by a large margin.","lead":"The paper introduces a benchmark that tests whether a vision model can locate the part of an object people interact with when the action word was never seen in training. It also presents a method combining an off-the-shelf detector with language-image similarity, reporting large gains over existing affordance grounding models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AGD20K-NAT and UMD-NAT appear to split at the image-affordance-pair level, so the same image may appear in both validation and test, confounding the reported 12.3% IoU@0.4 gain with image familiarity.","rationale":"The reader's weakest assumption is that the dependence on Grounded-SAM's detection (80.29% recall) is the main risk. Detection failure is real but is partially mitigated by the global-branch fallback (when no object is detected, alpha is set to 1) and is explicitly ablated. A more fundamental threat is the construction of the evaluation splits themselves. The paper describes decomposing each image annotation into multiple image-affordance pairs and then splitting those pairs by affordance type while using the entire AGD20K test set for both validation and test. Because a single AGD20K image typically has multiple affordance annotations, the same image can appear in both the validation and test sets with different affordance labels. The trainable model selects its final checkpoint and hyperparameters on the validation set, so validation-test image overlap gives the method access to test-image visual content during model selection. This would inflate the headline 12.3% improvement and would also affect the reported baseline comparisons, undermining the paper's central claim of novel-affordance generalization. The concrete test is simple and decisive: check image-ID overlap across splits, and if present, re-run the comparison under an image-disjoint split. Pending that check, the verdict should remain conditional, but the condition is now the integrity of the benchmark splits, not merely code release or detection recall.","tokens_in":21811,"tokens_out":10033,"duration_ms":98190,"concrete_test":"From the released splits (or reconstructed using AGD20K and UMD metadata), extract the underlying image IDs for every train, validation, and test image-affordance pair. Check whether any image ID appears in more than one of the three splits. If an image appears in both validation and test, re-split at the image level by assigning all affordance pairs of an image to a single split, then rerun Tables 1-3. The concern is settled if the 12.3% IoU@0.4 advantage over OOAL persists under the image-disjoint re-split; if the advantage shrinks or vanishes, the reported numbers are confounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that existing methods fail on novel affordance types while AffordAnything/AffordAnything+ succeed. That claim depends on the NAT splits actually isolating novel-affordance generalization from image-level familiarity. The appendix (Sec. A1) states: 'we decompose each image annotation into multiple image-affordance pairs' and 'for the validation and test data, we use the entire AGD20K test set in the Seen setting and further split it following the divided validation and test affordance types.' Since an AGD20K image carries annotations for multiple affordance types, this pair-level splitting can place the same image into both the validation and test sets (for example, one image with 'push' in validation and 'drink_with' in test). The trainable AffordAnything+ selects its checkpoint and hyperparameters on the validation set (Sec. A2: 'we select the best model based on the corresponding metrics on the validation set'). If validation and test share images, model selection can exploit the visual content of test images, inflating the reported 12.3% absolute IoU@0.4 improvement over OOAL. The paper does not report image-level disjointness or any filtering to prevent this. This is not a minor implementation detail: every quantitative comparison in Tables 1-6 is conditional on the splits being clean.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, zero-shot 2D grounding with novel affordance types (NAT), and introduces NAT splits of the AGD20K and UMD datasets. The authors propose AffordAnything, a training-free pipeline that detects objects with Grounded-SAM, decomposes the object masks into local patches, and scores the patches with CLIP features against the affordance text, fusing global and local branches. They also propose AffordAnything+, a trainable extension that learns a branch-weight predictor, a local weight bias, and a global branch decoder. On the proposed benchmarks the paper reports large gains over existing affordance grounding and open-vocabulary segmentation baselines, including a 12.3% absolute IoU@0.4 improvement over OOAL on AGD20K-NAT, and it claims that existing methods fail to generalize to novel affordance types while the proposed approach succeeds.","tokens_in":22061,"tokens_out":5623,"duration_ms":55418,"significance":"If the empirical claims hold, this paper would fill a genuine gap: prior 2D affordance grounding almost exclusively evaluates novel object categories with fixed affordance sets, while the NAT setting is closer to open-ended instructions in real applications. The proposed method is simple, modular, and grounded in an interpretable intuition about object subparts, and the appendix contains useful ablations and detection-recall analysis. The significance is, however, currently conditional on the benchmark being clean: the pair-level splitting described in Appendix A1 creates a real risk of image leakage between validation and test, and the central numbers in Tables 1-6 are exactly what would be affected. The paper also ships no code for the new splits, so the leak cannot be verified or ruled out by inspection.","major_comments":[{"comment":"The NAT splits are constructed at the image-affordance-pair level, not the image level. Appendix A1 states that 'For the validation and test data, we use the entire AGD20K test set in the Seen setting and further split it following the divided validation and test affordance types.' Because a single image can carry annotations for multiple affordance types, the same image can appear in both the validation set and the test set (and potentially in the training set) under this procedure. The main text only asserts that affordance types are disjoint across splits, which says nothing about image overlap. Since Appendix A2 states that AffordAnything+ selects the best model based on the validation set, shared images between validation and test mean that model selection can exploit the visual content of test images. This directly affects the credibility of the reported 12.3% absolute IoU@0.4 improvement over OOAL in Table 2 and the split-robustness claims in Table 3. The authors must either demonstrate image-level disjointness for all splits of AGD20K-NAT and UMD-NAT, or rebuild the splits so that no image appears in more than one split, and re-report all comparisons.","section":"Appendix A1, Sec. 5.1, Tables 1-3"},{"comment":"The paper reports that Grounded-SAM detects an object in 281 of 345 test samples and that object-level recall is 80.29%. This means that for roughly one fifth of test samples the local branch cannot contribute and the method falls back to a uniform mask. The method's central premise is that the affordance query in Eq. (1) localizes the relevant object; when that premise fails, the reported test metrics are driven by a different mechanism. The paper should report results conditioned on detection success, and should check whether the claimed advantage over baselines holds on the subset of samples where detection succeeds, as well as on the fallback subset.","section":"Appendix A6, 'Detection-and-segmentation backbone'"},{"comment":"All main quantitative results are reported for a single random training-sample selection with no error bars or multiple seeds. On AGD20K-NAT the one-shot training set contains only 63 image-affordance pairs, so the sampling of the 50 egocentric images per object category could substantially change the outcome. The absence of variance estimates is especially problematic because the paper's headline conclusion is that AffordAnything+ 'consistently' outperforms baselines; without repeated trials or seeds, consistency is asserted, not demonstrated. Please report means and standard deviations over at least five random training-sample selections (and multiple seeds for the trainable modules) for Tables 2 and 3.","section":"Sec. 5.3, Appendix A1"}],"minor_comments":[{"comment":"The notation f_t(f_I)^T / sqrt(d) is ambiguous: if f_t and f_I are both d-dimensional vectors, this expression is a scalar, not a spatial map. Please clarify how the global-branch prediction Y^G is formed from the image and text features, especially how a vector-level similarity becomes a pixel-level map.","section":"Eq. (8)"},{"comment":"The text encoder E_T is omitted from the diagram even though the text feature f_t is essential to the semantic fusion module; please add it or explicitly explain the omission in the caption.","section":"Figure 2"},{"comment":"The main text says 'the affordance types in the train, val, and test sets are mutually exclusive,' which is a statement about affordance types, not about images. Please state the actual split unit (image-affordance pair versus image) in the main text and make the appendix description consistent with it.","section":"Sec. 5.1, Appendix A1"},{"comment":"In the table, the 'Ours' row claims quantitative evaluation of novel affordance types in 2D, but no existing 2D baseline in the table has a checkmark in the 'Aff.' column. The table is useful as a summary, but the novelty claim would be easier to evaluate if the quantitative NAT results for 2D baselines were referenced alongside the table.","section":"Table A12"},{"comment":"AffordAnything+ has a total inference time of 0.680 s versus 0.021 s for OOAL on the same GPU. The paper does not discuss the runtime-accuracy tradeoff in the main text; a brief sentence noting this cost and in which applications it is acceptable would help practitioners judge the method.","section":"Sec. A8"},{"comment":"The description says the default branch weight alpha is set to 0 for AffordAnything, which means the global branch is not used at all; the later ablation in Table A8 shows that initializing alpha to 0 during training for AffordAnything+ performs much worse than initializing to 0.5. This difference between the training-free default and the trainable initialization is worth explaining more explicitly in the main text.","section":"Sec. A2, 'AffordAnything'"}],"recommendation":"major_revision","confidential_remarks":"The pair-level split issue is the most serious problem and will require the authors to rebuild or at least verify the splits at the image level. If the splits cannot be made image-disjoint without substantially changing the benchmark, the central comparison becomes hard to interpret. The paper otherwise has a plausible idea and useful ablations, so a rework with clean splits and error bars could make it publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the paper defines a genuinely new evaluation setting—zero-shot grounding to novel affordance types in 2D—and builds the first public benchmarks for it (AGD20K-NAT, UMD-NAT). The proposed pipeline, which detects the object with Grounded-SAM, crops local patches, and scores them with CLIP, is a sensible engineering combination and it beats existing affordance grounding and open-vocabulary segmentation baselines by a wide margin. Second, the headline number (12.3% absolute IoU@0.4 over OOAL) has a potential confound that needs to be resolved before the result is credible: the NAT splits appear to be made at the image-affordance-pair level, and the same image may appear in both validation and test. Since the trainable model selects its checkpoint on validation, this can leak test-set visual content.\n\nThe task formulation is the real contribution. Most prior work tests novel object categories with fixed affordances; here the objects are seen and the action words change. That's the right question for robot instruction following. The benchmarks are simple re-splits, but they are useful and the paper is careful to report two additional splits. The method itself is not exotic: detect, crop, score. But the local-patch decomposition of object masks is a reasonable inductive bias, and the ablations show each piece (evenly-focused patches, border patches, branch weighting) contributes. The authors are also candid about failure modes and about the detection backbone's 80% recall.\n\nWhere it gets soft. The split issue is the main one. The appendix says val/test both come from the entire AGD20K test set, with pairs assigned by affordance type. An image with multiple affordance annotations can easily land in both val and test. The paper never reports image-level disjointness. If there is overlap, the model selection performed on validation has effectively seen the test images, and the reported gains—large and consistent across methods—could be partially an artifact of image familiarity rather than affordance generalization. This is not a minor detail; every comparison in the paper rests on the splits being clean. The authors can fix it by reporting the overlap, or better, by filtering images so no image appears in more than one split.\n\nSecond, all the quantitative results are from a single run, with no seeds and no error bars. For a benchmark paper that others will build on, that's thin. The proposed method also gets to use Grounded-SAM (a strong open-vocabulary detector) while the baselines do not; the paper partially addresses this with ablations, but the main tables compare unequal toolkits. The large effect size makes these concerns survivable, but only if the splits are clean and the numbers reproducible.\n\nBottom line: the paper is worth engaging with seriously. It proposes a useful task, gives it a first benchmark, and offers a credible baseline. But I would not take the 12.3% claim at face value until the image-overlap question is resolved and the code and splits are released. This deserves peer review, with the split check as the first thing reviewers should demand.","headline":"Useful new benchmark for novel-affordance grounding, but the headline numbers rest on pair-level splits that may leak test images into validation.","tokens_in":22609,"tokens_out":4991,"would_cite":false,"duration_ms":45105,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot grounding of novel affordance verbs works by detecting the object with the verb and scoring its subpart patches with a frozen vision-language model.","keywords":["affordance grounding","novel affordance types","zero-shot generalization","2D grounding","vision-language models","open-vocabulary segmentation","object subpart patches","NAT benchmark"],"falsifier":"Present the pipeline with images where the same verb names no object (e.g., 'drink' on a sealed bottle with no cup) and check whether a plausible but wrong object mask is returned; if random substitute verbs produce equally good grounding masks, the affordance-specific reasoning is not doing the work. A simpler check already exists in the paper: compare the samples where the detector returns no object against a version that always returns the full-image mask, since the gap measures how much of the result rests on detection.","tokens_in":21580,"feed_emoji":"🖐️","tokens_out":8795,"duration_ms":70084,"temperature":0.7,"pith_summary":"Affordance grounding usually assumes the set of human–object interaction verbs is fixed at training time. This paper introduces the novel-affordance-types (NAT) setting, in which the test verbs are disjoint from training verbs, and argues that existing one-shot affordance methods and open-vocabulary segmentation models fail at it. The proposed training-free pipeline, AffordAnything, detects the object using the affordance verb as an open-vocabulary query and then scores refined subpart patches with a frozen vision-language model; the trainable AffordAnything+ learns only a small fusion module on top. On the re-split AGD20K-NAT benchmark, the best variant improves IoU@0.4 by 12.3 absolute percentage points over the prior state of the art, supporting the paper's claim that verb-level generalization is achievable without task-specific supervised training.","feed_headline":"Patch-and-score method grounds never-seen action words","feed_subtitle":"On a new benchmark, training-free grounding beats prior affordance models by 12.3 IoU points.","key_machinery":"The mechanism is the local-patch decomposition of object masks: each detected mask is cut into an even grid of interior patches plus a border-focused set, so affordance regions that sit on a subpart (handle, rim, edge, opening) become individual candidates. Cosine similarity between the verb's text embedding and each patch's visual embedding, softmaxed over patches, provides a training-free attention map; AffordAnything+ adds a small trainable fusion module with a branch-weight predictor, a local weight bias, and a cross-attention global decoder that are learned with one-shot supervision while the patch scoring stays grounded in the frozen model.","core_discovery":"The paper claims that zero-shot grounding of never-seen affordance types is possible by decomposition and scoring rather than by learning a fixed affordance vocabulary. Given an image and a novel verb, the method queries an open-vocabulary detector with the verb itself, refines each detected object mask into evenly focused and border-focused local patches, and uses cosine similarity between the verb's text embedding and each patch's visual embedding to build a soft affordance map. A global image-level branch is fused with the local branch, and in the trainable variant a lightweight decoder and learned fusion weights refine the combination. The paper reports that on AGD20K-NAT this pipeline reaches 21.3 IoU@0.4 in the training-free setting and 22.9 in the one-shot setting, against 10.6 for the strongest prior affordance model, and that the same pipeline also generalizes to novel object–novel affordance combinations.","pith_inferences":["If detection is the limiting factor, then supplying the object category as an auxiliary query (for example, the object named alongside the verb) should raise the roughly 80% detection recall and lift grounding further; the paper's own failure analysis implies this test.","The spatial-patch decomposition is a generic inductive bias for function-bearing subregions, so the same recipe might transfer to 3D affordance grounding or part-based robotic manipulation once a 3D mask decomposition replaces the 2D patches.","The learned branch weight likely encodes how much a verb is tied to object identity versus location; measuring the branch weight across verbs predicts that abstract verbs like 'use' lean more on the global branch than spatial verbs like 'open'."],"forward_implications":["Current closed-set affordance models do not merely underperform; they fail to transfer to unseen verbs, so new benchmarks are needed to measure verb-level generalization.","A training-free combination of open-vocabulary detection and patch scoring with a frozen vision-language model is enough for substantial gains, meaning the bottleneck is not task-specific training data.","The 12.3-point IoU@0.4 improvement on AGD20K-NAT, with consistent gains on UMD-NAT and on two additional splits, suggests the finding is stable across different ways of partitioning affordance types.","Because the local cue is affordance-independent, the same patch-scoring module can be reused for any new verb without retraining the backbone."],"supporting_citations":[{"why":"Supplies the AGD20K dataset, whose images and affordance labels are re-split into the proposed AGD20K-NAT benchmark.","marker":"[43]"},{"why":"Supplies the UMD part-affordance dataset, re-split as UMD-NAT in the paper.","marker":"[52]"},{"why":"The prior one-shot affordance grounding baseline whose training protocol is adopted and whose numbers are the comparison target.","marker":"[36]"},{"why":"The open-vocabulary detection-and-segmentation backbone that produces object masks queried by the affordance label.","marker":"[62]"},{"why":"Provides the frozen vision-language encoders used to extract image, patch, and text features.","marker":"[58]"},{"why":"Provides the image encoder used by the trainable variant for a fair architecture comparison.","marker":"[55]"},{"why":"A training-free open-vocabulary segmentation baseline compared in the training-free evaluation.","marker":"[5]"},{"why":"An open-vocabulary segmentation baseline used in both training-free and one-shot comparisons.","marker":"[33]"}],"fun_headline_variants":["Novel affordance grounding without training beats SOTA by 12.3","Zero-shot affordance grounding: decompose, score, win","AffordAnything: training-free grounding for unseen actions","New benchmark for zero-shot affordance grounding","Patch-based scoring grounds never-seen action words"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the affordance verb, used alone as a text query, will make the open-vocabulary detector find the object whose part is the target region; when detection fails, the local branch is silent, and the paper's appendix reports detection on only about 80% of test samples.","fun_headline_variants_meta":{"raw":{"variants":["Novel affordance grounding without training beats SOTA by 12.3","Zero-shot affordance grounding: decompose, score, win","AffordAnything: training-free grounding for unseen actions","New benchmark for zero-shot affordance grounding","Patch-based scoring grounds never-seen action words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2905,"prompt_tokens":882,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":498,"tokens_out":2023,"duration_ms":15563,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:19:37.962233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present the pipeline with images where the same verb names no object (e.g., 'drink' on a sealed bottle with no cup) and check whether a plausible but wrong object mask is returned; if random substitute verbs produce equally good grounding masks, the affordance-specific reasoning is not doing the work. A simpler check already exists in the paper: compare the samples where the detector returns no object against a version that always returns the full-image mask, since the gap measures how much of the result rests on detection.","supporting_citations":[{"cited_title":"Learning affordance grounding from exocentric images","cited_arxiv_id":null,"evidence_quote":"Supplies the AGD20K dataset, whose images and affordance labels are re-split into the proposed AGD20K-NAT benchmark."},{"cited_title":"Caldwell, and Nikos G","cited_arxiv_id":null,"evidence_quote":"Supplies the UMD part-affordance dataset, re-split as UMD-NAT in the paper."},{"cited_title":"One-shot open affordance learning with foundation models","cited_arxiv_id":null,"evidence_quote":"The prior one-shot affordance grounding baseline whose training protocol is adopted and whose numbers are the comparison target."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the frozen vision-language encoders used to extract image, patch, and text features."},{"cited_title":"Dinov2: Learning robust visual features without supervision.TMLR, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the image encoder used by the trainable variant for a fair architecture comparison."},{"cited_title":"ClearCLIP: Decomposing CLIP representations for dense vision-language inference","cited_arxiv_id":null,"evidence_quote":"An open-vocabulary segmentation baseline used in both training-free and one-shot comparisons."}],"review_version":1}