{"id":"d169abad-359c-4f99-a38d-245d0716fe0f","arxiv_id":"2507.17401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Wilhelm Tell is a 189-video, dual-perspective dataset of humans demonstrating household affordances such as stabbing, cutting, pouring, and stacking.","lead":"This paper presents Wilhelm Tell, a video dataset of 23 people performing small apple-based household tasks recorded from both a head-mounted and a frontal camera, with annotations of which affordances each task demonstrates. The dataset is offered as a resource for training robots to recognize affordances as events rather than static object features.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set annotations have no documented protocol or validation; without re-annotation agreement checks, the dataset's value as affordance training/evaluation ground truth is unsupported.","rationale":"The central claim is that the Wilhelm Tell dataset is a curated resource enabling affordance recognition training from video. The strongest verification of this claim would be a demonstration that a perception model trained on the dataset improves at recognizing affordance hotspots; the paper provides no such experiment. But even before that, the dataset's own annotations—the test-video CSV time intervals and hotspot masks—are the only human-verified ground truth, and their reliability is entirely unaddressed. Section VI describes their existence but not how they were created, what definitions were used, or any quality control. Since the proposed pipeline in the same section depends on automatic label generation for training, the test annotations are the sole check on correctness. If they are noisy or semantically inconsistent, the dataset cannot support the claimed use case, and any benchmark built on it would be misleading. The reader's weakest assumption identifies this same issue; re-annotation of a sample would settle it. The paper's other weaknesses—lack of baselines, unclear train/test split—are secondary and addressable, so the conditional verdict stands.","tokens_in":8009,"tokens_out":6924,"duration_ms":73910,"concrete_test":"Download the dataset from the GitHub repository (https://github.com/ease-crc/WilhelmTellDataset). Select a random sample of 10 test videos spanning all 20 tasks. Independently annotate, with two trained annotators following a written protocol that defines an interaction interval as the contiguous segment during which the named object physically contacts or manipulates the target object (e.g., fork tines in the apple for Stab). Compute temporal Intersection-over-Union (tIoU) between the two annotators and between each annotator and the provided CSV intervals. Also compare the provided hotspot masks on the extracted still frames against a re-annotation of the same frames (e.g., pixel IoU). If mean pairwise tIoU or hotspot IoU falls below 0.7, the annotations are not reliable enough to support the claimed training/evaluation use case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the test-video CSV time intervals and hotspot annotations are reliable enough to serve as ground truth for evaluating affordance recognition. Section VI states only that 'The test videos have corresponding CSV files describing time intervals in which objects interact to manifest an affordance. Some of the frames from the test videos have been extracted as still images and annotated with affordance hotspots for objects.' No annotation protocol, no inter-annotator agreement, and no validation are reported. The semantics of an 'interaction' interval are underspecified: for the Stab task, does the interval start at first contact, at full penetration, or when the fork is lifted? Without a written definition, intervals annotated by different people—or by the same person across sessions—may be inconsistent. This matters because these intervals and hotspots are the only human-verified labels in the dataset; the proposed training pipeline in Section VI relies on an external automatic labeling system, so the test annotations are the sole check on whether that system has learned the right affordance manifestations. If the intervals are noisy or misaligned, any model evaluated on them receives incorrect supervision signals, and the claim that the dataset 'can be used for robot training purposes' is unsupported. Additionally, no baseline experiment is provided to show that a model can learn affordance hotspots from the videos, so the dataset's utility is entirely unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Wilhelm Tell Dataset, a collection of 189 videos (417 minutes) of human demonstrations of 20 household tasks, mainly centered on apples, recorded from both ego and third-person perspectives. The dataset includes task-level affordance labels, a train/test split, and for test videos CSV interval annotations and some still frames with affordance hotspot annotations. The authors state the intended use is to train and evaluate visual perception systems for affordance recognition, with an envisioned pipeline that relies on an external automatic hotspot labeling system. The paper describes the study setup, participant recruitment and consent, task lists, and usage notes, and the dataset is released under CC-BY 4.0.","tokens_in":8201,"tokens_out":3102,"duration_ms":35509,"significance":"If validated, the dataset would fill a notable gap by providing a video-based alternative to static-image affordance datasets, with multiple perspectives and an emphasis on affordances manifested through object-object and agent-object interactions. The inclusion of varied task performances and preparatory maneuvers could support research on task-space organization and collaborative robots. The paper also documents ethical procedures, GDPR-compliant storage, participant consent, and a public license, which are strengths. However, the contribution is currently unvalidated: the paper provides no annotation protocol, inter-annotator agreement measures, or baseline experiments, so the central usability claim is not yet supported.","major_comments":[{"comment":"The test-video CSV time intervals and hotspot annotations are the only human-verified labels in the dataset, but no annotation protocol is described and no inter-annotator agreement is reported. For example, for the Stab task it is unspecified whether the interaction interval starts at first contact, at full penetration, or at the lift of the fork. Without a precise semantic definition of an 'interaction' interval, annotations may be inconsistent across annotators and across sessions, undermining their validity as ground truth for the proposed evaluation pipeline.","section":"VI"},{"comment":"The paper claims the dataset is 'aimed towards training perception systems to recognize affordance manifestations' and 'can be used for robot training purposes,' yet no experiment or proof-of-concept demonstrates that the dataset actually supports such training. The envisioned use relies on an external automatic labeling system (e.g., [29]) that is not evaluated here. At least a baseline experiment, even a simple one, or quantitative evidence of annotation quality (e.g., coverage statistics, agreement) is needed to substantiate the usability claim.","section":"VI"},{"comment":"There is a numerical inconsistency in the dataset description: Section V reports 189 total videos, while Section VI states the videos are split into 102 training and 44 testing videos, which sums to 146. The paper does not explain how the remaining 43 videos, the combined per-perspective videos, and the incomplete participant entries relate to the train/test split. This ambiguity affects reproducibility and the interpretation of the dataset's composition.","section":"V and VI"},{"comment":"The mapping between tasks and affordance labels is not specified in a machine-readable schema. For instance, Task 4 in Table I is associated with two source labels (Contain-ability from AfNet and Contain from AffordanceNet), and it is unclear whether the CSV files contain only task identifiers or also the affordance labels derived from these sources. A clear metadata schema or example annotation file would significantly improve the dataset's usability.","section":"IV and V"}],"minor_comments":[{"comment":"The text contains several typos, including 'afordance' instead of 'affordance', 'demonstartion' instead of 'demonstration', and 'developement' instead of 'development'.","section":"II"},{"comment":"There are typos in the methods section: 'headstreap' should be 'headstrap', 'corrent' should be 'correct', and 'layed out' should be 'laid out'.","section":"IV"},{"comment":"The sentence 'The Readme file for the dataset containing information as well as a download link to the video material can be found on GitHub' is awkwardly phrased; consider rewriting for clarity and including the direct repository URL in the text.","section":"V"},{"comment":"The word 'automtically' is a typo for 'automatically' in the description of the labeling pipeline.","section":"VI"},{"comment":"The explanation of the session ordering (tasks 8-10, then 1-7, then 11-20, with a/b variants) is somewhat confusing; a timeline or flow diagram of the bundles would help readers understand the recording structure.","section":"IV"}],"recommendation":"major_revision","confidential_remarks":"The dataset has potential, but the manuscript currently lacks the validation evidence that a dataset paper needs. The annotation protocol and consistency checks are the most pressing issues; without them, the dataset cannot serve as ground truth. A baseline demonstration of trainability would also strengthen the contribution significantly. The numerical inconsistency in video counts should be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a decent, honest dataset paper, and the artifact is genuinely new. Wilhelm Tell collects 189 videos (about seven hours) of 23 people doing 20 apple-centered household tasks, with paired GoPro first-person and static third-person views. That pairing plus the focus on multi-object affordances (Stab, Pour, flow support, etc.) distinguishes it from EPIC-KITCHENS and Ego4d, which are mostly egocentric and not affordance-annotated. The task design is thoughtful: broad instructions, task variants, and object swaps generate natural variation in preparation and execution. Consent and GDPR handling are described. Related work is cited sensibly.\n\nThe soft spots are real, and the stress-test note lands. Section VI says test videos have CSV time intervals \"in which objects interact to manifest an affordance\" and some frames have hotspot annotations, but there is no annotation protocol, no inter-annotator agreement, and no validation. The semantics of an interval are underspecified; for the Stab task, does it start at contact, full penetration, or lift? Since the proposed training pipeline relies on an external automatic labeling system, those test annotations are the only check on whether the system learned the right affordance manifestations. Without validation, the claim that the dataset supports affordance-recognition training is unsupported. The paper also reports no baseline experiments, so we don't know whether the videos even support learning hotspots.\n\nThese problems are addressable. The collection procedure, dataset statistics, and intended use are clear; what's missing is annotation documentation and a minimal evaluation. The access route is indirect too: the Readme on GitHub points to university servers, which is fine in principle but should be made stable and direct.\n\nFor a dataset introduction, that's enough to merit peer review. A serious referee can require the annotation protocol, agreement numbers, and at least one baseline before acceptance. I wouldn't desk-reject. I also wouldn't cite it in my own work until the labels are documented, but I'd likely bring it to a reading group. It's a useful resource for affordance-from-video researchers, and the flaws are proportional; they don't sink the underlying value.","headline":"A genuinely new paired first/ and third-person affordance video dataset, but the unvalidated test annotations and missing baselines leave the intended use undemonstrated.","tokens_in":8727,"tokens_out":2125,"would_cite":false,"duration_ms":25195,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents a curated set of 189 dual-perspective videos of everyday household tasks, labelled with the affordances they manifest, so that robots can learn to recognize possibilities for action from observed events rather than…","keywords":["affordance demonstrations","robot learning from observation","video dataset","household tasks","egocentric and third-person video","affordance recognition","semantic segmentation"],"falsifier":"Compute inter-annotator agreement on the hotspot still frames and check whether the annotated time intervals align with frames in which the object is actually grasped, lifted, or otherwise used; if agreement is near chance or the intervals frequently miss the interaction, then the dataset cannot support the intended supervision.","tokens_in":7820,"feed_emoji":"🎥","tokens_out":5462,"duration_ms":56391,"temperature":0.7,"pith_summary":"This paper presents the Wilhelm Tell dataset, a curated collection of 189 videos totalling about seven hours of people performing 20 everyday tasks centered on apples and household objects, recorded simultaneously from a first-person head camera and an external frontal camera. The authors' aim is to provide training and evaluation data for perception systems that recognize affordances — possibilities for action — as they are manifested in real events, rather than from static images of object parts. Each task is labelled with the affordance it demonstrates (drawn from existing shape datasets, plus 'Stab' and 'Pour'), and the test videos come with CSV time-interval annotations describing when object interactions happen, some with hotspot-annotated still frames. The intended use is to train automatic affordance-labeling systems and then semantic segmentation models that detect functional object parts both during and outside moments of use.","feed_headline":"189 videos train robots to spot action possibilities","feed_subtitle":"Everyday tasks filmed from two cameras, with time-stamped labels, replace static images in affordance learning.","key_machinery":"The central object is the dataset itself: 189 MP4 videos recorded with a GoPro headset (ego perspective) and a frontal camera (third-person perspective), grouped into bundles of tasks 1–20 and accompanied by metadata linking each task to an affordance label such as Cut, Contain, Stab, Pour, Stack, Wrap, Filter, or Connect. The 'affordance manifestation' is the mechanism: each video shows an event in which an object's functional part is actually used, and the test split additionally provides CSV time intervals marking when objects interact and hotspot-annotated frames marking where the affordance is located. These annotations make the videos usable as supervision for an automatic affordance-hotspot labeling system, which in turn can train a semantic segmentation model to recognize functional parts both in use and not in use.","core_discovery":"The central claim is that affordances are better learned through video demonstrations of their manifestation than through static shape annotations, and that a dataset of such demonstrations is a reusable resource for robot training. Concretely, the paper contributes 189 MP4 videos (417 minutes) from 23 participants, with task-level affordance labels, a train/test split of 102/44 videos, test-video CSV files marking time intervals of object interactions, and still frames with affordance hotspots for some test frames. The authors deliberately gave participants only broad instructions, so task performances vary in preparation maneuvers and object placement; this variety is itself part of the contribution, enabling study of how people arrange task space. The paper positions the dataset between ego-centric video collections and static-shape benchmarks by adding an observer perspective and a focus on affordances involving multiple objects interacting.","pith_inferences":["The absence of any reported annotation protocol, inter-annotator agreement, or baseline evaluation in the paper means the dataset should initially be read as a set of candidate labels; establishing benchmark quality would require an external reproducibility study.","The a/b task variants with different object sets create a ready-made domain-shift test: a model trained on one variant can be evaluated on the other to check whether it learned the affordance itself rather than features of a particular object.","At roughly seven hours of video, the dataset is small by current standards, but its dense annotation of demonstrations could serve as a seed set for bootstrapping larger automatically labeled video corpora for affordance learning.","If the preparatory maneuvers observed in the recordings were later annotated systematically, the same videos would double as a resource for studying spatial arrangement and expected action sequences, not only affordance hotspots."],"forward_implications":["Using the 102 training and 44 test videos, a researcher can train an automatic affordance-hotspot labeling system and then use the generated labels to train an image segmentation model that recognizes functional parts during and outside moments of use.","Because every task is recorded from both ego and frontal views, methods can learn from either perspective or exploit cross-view consistency between them.","The twenty tasks cover affordances that involve multiple objects interacting with each other and the agent, such as Stack, Wrap, Filter, and Connect, going beyond the agent-object interactions covered by earlier video affordance datasets.","The deliberate lack of prescribed preparation procedures means the same recordings can support research on preparatory maneuvers and space organization, informing collaborative robots that anticipate human actions."],"supporting_citations":[{"why":"Gibson 1979; provides the original definition of affordance as what the environment provides to an agent, which motivates the dataset's event-based framing.","marker":"[16]"},{"why":"AfNet; one of the two static shape-part datasets from which the affordance labels are drawn.","marker":"[23]"},{"why":"3D AffordanceNet; the other static benchmark whose affordance classes the dataset selects and extends with Stab and Pour.","marker":"[24]"},{"why":"Mur-Labadia et al.; supplies the automatic affordance-hotspot labeling pipeline that the dataset is designed to train and evaluate.","marker":"[29]"},{"why":"Luo et al.; prior work learning visual affordance grounding from demonstration videos, establishing the video-based approach the dataset supports.","marker":"[30]"},{"why":"Demo2vec; the comparison dataset whose smaller affordance set the work extends toward multi-object interactions.","marker":"[31]"},{"why":"EPIC-Kitchens; an ego-centric video dataset compared with this work because it lacks the extra observer perspective.","marker":"[32]"},{"why":"Ego4D; another ego-centric video collection whose single-perspective design the dataset contrasts with its dual views.","marker":"[33]"}],"fun_headline_variants":["189 videos show robots the possibilities for action","Video dataset for affordances: 189 clips, 7 hours","First- and third-person task videos for robot affordance learning","From static shapes to videos: new affordance dataset","Robot affordance training: 189 task videos from 23 people"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test-video CSV time intervals and the hotspot annotations on selected still frames are accurate and consistent enough to serve as ground truth for training and evaluating affordance recognition systems.","fun_headline_variants_meta":{"raw":{"variants":["189 videos show robots the possibilities for action","Video dataset for affordances: 189 clips, 7 hours","First- and third-person task videos for robot affordance learning","From static shapes to videos: new affordance dataset","Robot affordance training: 189 task videos from 23 people"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1465,"prompt_tokens":832,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":561}},"tokens_in":448,"tokens_out":633,"duration_ms":9108,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:47:31.080756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute inter-annotator agreement on the hotspot still frames and check whether the annotated time intervals align with frames in which the object is actually grasped, lifted, or otherwise used; if agreement is near chance or the intervals frequently miss the interaction, then the dataset cannot support the intended supervision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gibson 1979; provides the original definition of affordance as what the environment provides to an agent, which motivates the dataset's event-based framing."},{"cited_title":"Afnet: The affordance network,","cited_arxiv_id":null,"evidence_quote":"AfNet; one of the two static shape-part datasets from which the affordance labels are drawn."},{"cited_title":"3d affordancenet: A benchmark for visual object affordance understanding,","cited_arxiv_id":null,"evidence_quote":"3D AffordanceNet; the other static benchmark whose affordance classes the dataset selects and extends with Stab and Pour."},{"cited_title":"Multi-label affordance mapping from egocentric vision,","cited_arxiv_id":null,"evidence_quote":"Mur-Labadia et al.; supplies the automatic affordance-hotspot labeling pipeline that the dataset is designed to train and evaluate."},{"cited_title":"Learning visual affordance grounding from demonstration videos,","cited_arxiv_id":null,"evidence_quote":"Luo et al.; prior work learning visual affordance grounding from demonstration videos, establishing the video-based approach the dataset supports."},{"cited_title":"Demo2vec: Reasoning object affordances from online videos,","cited_arxiv_id":null,"evidence_quote":"Demo2vec; the comparison dataset whose smaller affordance set the work extends toward multi-object interactions."},{"cited_title":"The epic- kitchens dataset: Collection, challenges and baselines,","cited_arxiv_id":null,"evidence_quote":"EPIC-Kitchens; an ego-centric video dataset compared with this work because it lacks the extra observer perspective."},{"cited_title":"Ego4d: Around the world in 3,000 hours of egocentric video,","cited_arxiv_id":null,"evidence_quote":"Ego4D; another ego-centric video collection whose single-perspective design the dataset contrasts with its dual views."}],"review_version":1}