{"id":"82b9c965-aeb1-4d70-a427-7d29ce3ad810","arxiv_id":"2508.17922","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EIVA couples manipulation instructions to contact regions and 3D motion directions, and an Actor-Verifier LMM loop improves instruction-conditioned affordance prediction in egocentric scenes.","lead":"The paper introduces EIVA, a new egocentric dataset of 14,861 image-instruction-affordance triplets, and an iterative 'search against verifiers' pipeline in which one large multimodal model both proposes and verifies where to contact an object and which way to move it. The method improves zero-shot contact-region grounding over prior affordance models, but motion-direction gains are small and the dataset and code are not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Motion-direction ground truth lacks egomotion compensation in moving-camera egocentric video; reported CS values are tiny and no random or majority baseline is given.","rationale":"The reader's weakest assumption was the reliability of the EIVA ground-truth annotation pipeline, especially the documented failure modes without frequency or inter-annotator agreement. I agree that this is the most load-bearing area, but I would sharpen it to a specific, currently missing technical step: the motion-direction labels are computed from 3D trajectories in camera space without compensating for ego-motion in videos that are inherently egocentric. This is not merely an annotation-quality issue; it is a potential systematic bias in half of the affordance representation. It is also corroborated by the paper's own statements: SpatialTracker outputs are in 'camera space,' the trajectory 'cannot directly serve as ground truth,' and the CS acceptability criterion is only 'at least greater than 0.' The reported CS values are extremely small, and without a random or majority baseline they do not establish 'outstanding performance' for motion direction. The contact-region results and the AGD20K generalization experiment provide some independent support for the central claim, and the SAPIEN trials add a small amount of real-task validation, but none of these validate the motion labels themselves. I would keep the CONDITIONAL verdict rather than escalate to REJECT, because the flaw is identifiable and potentially fixable by recomputing labels with ego-motion compensation and reporting reliability statistics; however, until that is done, the motion-direction component of the benchmark and the reflection pipeline's claimed improvement on it remain unverified.","tokens_in":23026,"tokens_out":10360,"duration_ms":118775,"concrete_test":"Select a random subset of 100-200 EIVA triplets from Ego4D/EPIC-Kitchens and recompute the motion-direction labels after compensating for camera ego-motion: estimate camera motion from static background points (e.g., via SpatialTracker on background pixels or an off-the-shelf VO method), subtract it from the hand/object trajectories, then re-run the PCA/discretization. Independently, have two annotators label the 26-direction motion for the same clips. Then recompute CS for Qwen2.5-VL-7B-Reflection and GPT4o-SoM-Reflection against original vs. corrected labels, and compare with a uniform-random baseline and a majority-direction baseline. If the corrected labels substantially change the CS values or the method ranking, the reported motion-direction results are not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Motion Direction Extraction' defines ground-truth motion labels as the PCA principal direction of SpatialTracker trajectories in 'camera space' (3D positions in the camera frame), then discretizes to 26 directions. Ego4D and EPIC-Kitchens are egocentric videos with substantial head and body motion; the camera-frame trajectory of a hand/object is the sum of the manipulation motion and the camera's own motion. The paper describes no step to subtract camera ego-motion (e.g., from static background points or visual odometry), so labels such as 'backward/up/left' may reflect where the camera moved, not where the object should move for the task. The appendix lists four annotation failure modes but does not mention this one, and no frequency or inter-annotator agreement is reported for any failure mode. On the evaluation side, the Performance Metrics appendix explicitly sets an acceptability bar of CS 'at least greater than 0'; with the 26 symmetric discrete directions, a uniform random prediction has expected cosine 0, and a trivial majority-direction baseline could plausibly exceed the reported values (Table 1: 0.012–0.060; Table 2: 0.008–0.235). No such baseline, confidence interval, or per-category breakdown is provided. The SAPIEN trials give some independent support, but only 60 trials and no motion-label validation. Because motion direction is half of the proposed affordance representation, the central claim that the method 'unlocks' instruction-oriented affordance prediction and that the reflection pipeline improves it is not yet supported for the MD component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that affordance should be instruction-dependent rather than a static object property, and introduces EIVA, a dataset of 14,861 egocentric object-instruction-affordance triplets with contact-region and motion-direction annotations. It also proposes a \"search against verifiers\" pipeline in which an LMM (Qwen2.5-VL or GPT-4o) iteratively proposes and verifies affordances, and evaluates the approach zero-shot on EIVA, on AGD20K, and in SAPIEN simulations, together with a fine-tuned LISA-13B comparison. The central claim is that the method unlocks new instruction-oriented affordance prediction capabilities and outperforms prior affordance methods on contact-region localization, while motion-direction accuracy is positive but small.","tokens_in":23359,"tokens_out":5325,"duration_ms":52773,"significance":"If the EIVA annotations are reliable, the dataset is a useful benchmark for instruction-conditioned affordance, and the reflection pipeline provides a competitive zero-shot baseline. The paper is commendable for the breadth of its evaluation, including the hand-removal check, the AGD20K generalization test, the fine-tuned LISA comparison, and the transparent documentation of four annotation failure modes. However, the motion-direction evidence is currently too weak to support the paper's central claim that the method 'unlocks' instruction-oriented affordance prediction; the contact-region results are more convincing, but the motion-direction results (CS values of 0.008-0.235) require stronger statistical grounding and a check for egomotion bias.","major_comments":[{"comment":"The ground-truth motion directions are computed as the PCA principal direction of SpatialTracker trajectories in camera space, with no explicit subtraction of egocentric camera motion. Since Ego4D and EPIC-Kitchens are moving-camera videos, the reported directions such as 'backward/up/left' may reflect head/body motion rather than the object's task-relevant displacement. The appendix's failure-mode list does not include this issue, and the reported CS values in Tables 1 and 2 (0.008-0.235) are extremely close to the zero correlation expected from a random discrete direction under the paper's 26-way symmetric discretization. The appendix's statement that 'an acceptable result is that this metric is at least greater than 0' is not a sufficient statistical bar. Please add an egomotion-compensation step or quantify its effect on a validation subset, and report random and majority baselines with confidence intervals and per-category CS breakdowns.","section":"Motion Direction Extraction and Performance Metrics (Appendix)"},{"comment":"The paper states that all EIVA samples are manually verified to guarantee correctness, but the same appendix identifies four systematic failure modes (wrong active hand side, active hand not visible, wrong contact frame, noisy homography) and does not report how often each occurs or any inter-annotator agreement. Because these failure modes directly corrupt contact-region and motion-direction labels, the benchmark's validity and the conclusion that reflection improves affordance prediction depend on knowing their frequency and whether the manual review actually corrected them. Please report the fraction of samples affected by each failure mode, the review protocol, and inter-annotator agreement on a sample.","section":"Correctness of Automatic Annotation (Appendix)"},{"comment":"The success rates are reported without trial counts, confidence intervals, or per-instance data; with ten object categories, three instances each, and two tasks per instance, the total of 60 trials leaves very few trials per reported cell, so the 'consistent lift' from reflection is not statistically established. Please report the exact number of trials per method-object cell and per-task, and provide confidence intervals or raw trial outcomes.","section":"Performance in Robot Simulation Trails (Section and Table 4)"},{"comment":"The reflection pipeline is compared against a no-reflection single-proposal baseline, but there is no control that samples the same number of proposals without verifier feedback. The observed gains could partly reflect additional test-time sampling rather than the verifier-informed refinement loop; a baseline that generates T independent proposals and selects one at random (or by a fixed rule) would isolate the contribution of the verifier.","section":"LMM-based Affordance Prediction (Meta Framework) and Table 1"}],"minor_comments":[{"comment":"The caption of Figure 4 contains garbled unicode sequences that should be cleaned.","section":"Figure 4"},{"comment":"'Robot Simulation Trails' should be 'Robot Simulation Trials'.","section":"Section title"},{"comment":"The phrase 'relays on the assumption' should be 'relies on the assumption'.","section":"Correctness of Automatic Annotation"},{"comment":"The text contains the typo 'beacause' and the hyperparameter table lists 'Betas [0.9, 095]', which appears to be a typo for [0.9, 0.95].","section":"Details of Finetuning LISA"},{"comment":"Several cross-references in the appendix are unresolved (e.g., 'in Section') and should be replaced with concrete section numbers.","section":"Appendix cross-references"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's dataset and benchmark contribution are potentially valuable, and the contact-region results are reasonably strong. However, the motion-direction evaluation is not yet at the standard required for the central claims; the absence of chance-level baselines, confidence intervals, and egomotion compensation makes it impossible to assess whether the model's motion-direction predictions are meaningful. I would advise the editor to require the baseline analysis and annotation-reliability report before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth engaging with. It introduces the first egocentric, instruction-conditioned affordance dataset (14,861 triplets, 146 object-instruction combos) and a zero-shot LMM baseline built on an Actor-Verifier reflection loop. The central claim—that affordance should be task-dependent rather than a static object property—is well argued and the dataset design follows from it cleanly. The annotation pipeline is thoughtful: contact points are projected back to pre-contact frames to avoid occlusion, motion directions are 3D and discretized into 26 directions, and the appendix documents four failure modes with visualizations. That is real work, and it gives the field a reusable benchmark if the data is released.\n\nThe reflection pipeline also mostly holds up. Both Qwen2.5-VL and GPT-4o improve with reflection on contact region metrics, and the improvement transfers to AGD20K and to the SAPIEN trials. The ablation showing that visualizing proposals matters is a good check, and the hand-removal experiment helps rule out the obvious shortcut. These are the right experiments.\n\nThe soft spots are real but localized. The biggest one is motion direction. The stress-test note about egomotion is worth taking seriously: the paper says trajectories are in \"camera space\" but never specifies whether that means a fixed first-frame coordinate system or per-frame camera coordinates. If it is per-frame, the missing egomotion compensation would contaminate the labels. The reported CS values are tiny (0.008 to 0.235), and the paper sets the acceptability bar at \"at least greater than 0\" without giving a random or majority baseline. That is a weak evidential standard. The SAPIEN results provide some independent support for the direction signal, but with only 60 trials and no error bars, that support is thin. Second, the dataset and code are not released, which limits reproducibility of the central benchmark claim. Third, annotation reliability is asserted but not quantified: no inter-annotator agreement or failure-mode frequencies, though the manual review claim is plausible.\n\nOverall, the contact-region half of the paper is solid and the motion-direction half is under-supported. I would not desk-reject this. It deserves a serious referee and likely a conditional accept: release the dataset, clarify the egomotion issue or compensate for it, add baselines and confidence intervals for CS, and report trial counts and annotation reliability.","headline":"EIVA is a genuinely useful new task and dataset with a plausible but under-supported motion-direction component; worth refereeing, conditional on releasing the data and answering the egomotion question.","tokens_in":23915,"tokens_out":2401,"would_cite":false,"duration_ms":28526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that affordance should be instruction-dependent, and that one large multimodal model can predict it by proposing contact regions and motion directions and then verifying its own proposals in an iterative loop.","keywords":["affordance prediction","instruction-oriented affordance","egocentric vision","large multimodal model","self-refinement","robot manipulation","EIVA dataset","contact region grounding"],"falsifier":"Independently re-annotate a random sample of EIVA triplets with fresh human labelers who see only the instruction and the pre-contact image, then recompute the reflection-versus-baseline score differences on the corrected labels. If the differences disappear or reverse, the central claim that the Actor-Verifier loop improves instruction-oriented affordance prediction is falsified; if the annotation failure modes appear in a large fraction of samples, the benchmark itself is in question.","tokens_in":22834,"feed_emoji":"🤖","tokens_out":9152,"duration_ms":83124,"temperature":0.7,"pith_summary":"The paper argues that an object's affordance is not a fixed property: the same drawer, door, or kettle offers different contact places and motion directions depending on whether the instruction is to open, close, pick up, or push. To make this claim concrete, it introduces EIVA, a dataset of 14,861 egocentric image-instruction-affordance triplets whose annotations come from an automated pipeline followed by manual review. On this benchmark, the paper shows that large multimodal models can act as instruction-oriented affordance predictors, and that an iterative \"search against verifiers\" loop, where one LMM proposes an affordance and then verifies and refines its own proposal, outperforms direct prompting and prior affordance methods. If correct, this gives robot manipulation a task-conditioned, open-vocabulary way to decide where and how to act without task-specific training.","feed_headline":"Instruction, not object, decides where robots should act","feed_subtitle":"A 14,861-scene egocentric dataset and an Actor-Verifier loop improve zero-shot contact and motion prediction.","key_machinery":"The central object is the \"search against verifiers\" loop: an Actor LMM generates an affordance proposal, and a Verifier LMM, the same model in a different prompt role, inspects a visualization of that proposal and either approves it or returns corrective feedback, after which the Actor refines and the loop repeats, with a final best-proposal selector. Two visualization mechanisms carry the argument: a SAM-generated mask of the proposed bounding box for the coordinate-based variant, and Set-of-Mark overlays that partition the image into labeled candidate regions for the closed-source model variant. The loop runs in two stages, first fixing the contact region and then fixing a 26-way discrete 3D motion direction, so verification is applied to each decision separately. A second load-bearing mechanism is the EIVA annotation pipeline: peripheral hand-edge points in the contact frame are projected backward through homographies to pre-contact frames to define occlusion-free contact regions, and SpatialTracker's 3D trajectories are summarized by PCA into one principal motion direction.","core_discovery":"The central claim is that affordance should be instruction-dependent: different tasks applied to the same object can require different contact regions and different post-contact motion directions, so affordance prediction should take a natural-language instruction as an input rather than reading off static object properties. The paper substantiates this with EIVA, a 14,861-triplet egocentric dataset spanning 146 object-instruction combinations, and with a zero-shot prediction pipeline in which one large multimodal model plays both Actor and Verifier. The Actor proposes a contact region and then a motion direction; the Verifier, prompted with a visualization of the proposal, judges it against the instruction, returns explanations and suggested alternatives when it is wrong, and the loop repeats until approval or a step limit. Across EIVA, the reflection pipeline beats direct LMM prompting and prior affordance methods on contact-region metrics and yields positive, though small, motion-direction cosine similarity; it also generalizes zero-shot to an existing affordance benchmark and improves success rates in robot simulation trials. The paper concludes that the new task, the dataset, and the reflection baseline together open a practical route to instruction-conditioned robot manipulation.","pith_inferences":["If the verifier's advantage comes mainly from correcting gross localization errors, the same loop could transfer to other dense prediction tasks where LMMs can critique but not generate precise outputs, such as grasp poses or part segmentation.","The benchmark's reliability hinges on the unquantified annotation failure modes; measuring their frequency with a fresh human annotation study would tell whether the reported performance gaps are genuine or partly label artifacts.","Because the motion-direction scores are small, a natural next step is to replace the 26-direction discretization with continuous or object-relative directions and check whether reflection still adds value.","The instruction-dependent view implies that a single image has multiple valid ground-truth affordances for different instructions, so evaluating models against all instructions on the same object, not just the annotated one, would directly measure how well a model disambiguates intent."],"forward_implications":["An egocentric robot can turn a single LMM into a task-conditioned affordance predictor by prompting it to propose, verify, and refine its own contact regions and motion directions.","Instruction-conditioned affordance data of the EIVA kind lets open-vocabulary models outperform methods that treat affordances as static object categories, especially in cluttered real-world scenes.","The reflection gains transfer zero-shot to existing affordance benchmarks and to simulated manipulation, so the loop can serve as a usable default baseline for instruction-oriented affordance prediction.","Fine-tuning a general-purpose multimodal segmentation model on EIVA unlocks affordance reasoning on unseen object categories, showing the dataset can support training as well as evaluation.","Motion-direction prediction is the harder part of the task and remains the main open target, with positive but small cosine-similarity scores."],"supporting_citations":[{"why":"Supplies the peripheral contact-point sampling and homography back-projection technique and serves as the main prior baseline.","marker":"(Bahl et al. 2023)"},{"why":"Established automatic affordance extraction from egocentric videos and is the predecessor definition that the paper contrasts with instruction-dependent affordance.","marker":"(Liu et al. 2022a)"},{"why":"Provides the Ego4D videos used for the real-world subset of EIVA.","marker":"(Grauman et al. 2022)"},{"why":"Provides the EPIC-KITCHENS videos used for the real-world subset of EIVA.","marker":"(Damen et al. 2021)"},{"why":"Provides the HOI4D videos used for the laboratory subset of EIVA.","marker":"(Liu et al. 2022b)"},{"why":"The hand-object detector that locates interactions in the annotation pipeline and whose hand-side errors are a documented failure mode.","marker":"(Shan et al. 2020)"},{"why":"SpatialTracker generates the 3D trajectories that feed the PCA step defining motion-direction ground truth.","marker":"(Xiao et al. 2024)"},{"why":"SAM generates masks that visualize Actor proposals for the Verifier and underlies the Set-of-Mark variant.","marker":"(Kirillov et al. 2023)"},{"why":"Set-of-Mark prompting enables candidate-region selection and visual feedback in the closed-source model variant.","marker":"(Yang et al. 2023)"},{"why":"Provides the agent-centric affordance baseline and the robot simulation protocol that the paper extends to task-oriented success.","marker":"(Li et al. 2024c)"}],"fun_headline_variants":["Why robots need instructions to know where to touch","Task text, not object shape, guides robot hands","Robot affordance now follows your command, not the object","New egocentric dataset teaches robots instruction-aware touching","Large model with self-verification improves robot manipulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated EIVA labels, contact regions from homography-projected hand-edge points and motion directions from tracked 3D trajectories, really capture the correct place and direction for each instruction, even though the paper documents four annotation failure modes and does not report how often they occur.","fun_headline_variants_meta":{"raw":{"variants":["Why robots need instructions to know where to touch","Task text, not object shape, guides robot hands","Robot affordance now follows your command, not the object","New egocentric dataset teaches robots instruction-aware touching","Large model with self-verification improves robot manipulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1262,"prompt_tokens":934,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":253}},"tokens_in":550,"tokens_out":328,"duration_ms":3618,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:59:51.450592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample of EIVA triplets with fresh human labelers who see only the instruction and the pre-contact image, then recompute the reflection-versus-baseline score differences on the corrected labels. If the differences disappear or reverse, the central claim that the Actor-Verifier loop improves instruction-oriented affordance prediction is falsified; if the annotation failure modes appear in a large fraction of samples, the benchmark itself is in question.","supporting_citations":[],"review_version":1}