REVIEW 4 major objections 5 minor 35 references
The Wilhelm Tell Dataset of Affordance Demonstrations
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper presents a curated set of 189 dual-perspective videos of everyday household tasks, labelled with the affordances they manifest, so that robots can learn to recognize possibilities for action from observed events rather than…
desk verdict A genuinely new paired first/ and third-person affordance video dataset, but the unvalidated test annotations and missing baselines leave the intended use undemonstrated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dataset itself: 189 MP4 videos recorded with a GoPro headset (ego perspective) and a frontal camera (third-person perspective), grouped into bundles of tasks 1–20 and accompanied by metadata linking each task to an affordance label such as Cut, Contain, Stab, Pour, Stack, Wrap, Filter, or Connect. The 'affordance manifestation' is the mechanism: each video shows an event in which an object's functional part is actually used, and the test split additionally provides CSV time intervals marking when objects interact and hotspot-annotated frames marking where the affordance is located. These annotations make the videos usable as supervision for an automatic affordance-hotspot labeling system, which in turn can train a semantic segmentation model to recognize functional parts both in use and not in use.
What would settle it
Compute inter-annotator agreement on the hotspot still frames and check whether the annotated time intervals align with frames in which the object is actually grasped, lifted, or otherwise used; if agreement is near chance or the intervals frequently miss the interaction, then the dataset cannot support the intended supervision.
Extended reading notes
Core claim
The central claim is that affordances are better learned through video demonstrations of their manifestation than through static shape annotations, and that a dataset of such demonstrations is a reusable resource for robot training. Concretely, the paper contributes 189 MP4 videos (417 minutes) from 23 participants, with task-level affordance labels, a train/test split of 102/44 videos, test-video CSV files marking time intervals of object interactions, and still frames with affordance hotspots for some test frames. The authors deliberately gave participants only broad instructions, so task performances vary in preparation maneuvers and object placement; this variety is itself part of the contribution, enabling study of how people arrange task space. The paper positions the dataset between ego-centric video collections and static-shape benchmarks by adding an observer perspective and a focus on affordances involving multiple objects interacting.
Load-bearing premise
The test-video CSV time intervals and the hotspot annotations on selected still frames are accurate and consistent enough to serve as ground truth for training and evaluating affordance recognition systems.
Editorial extensions
If this is right
- Using the 102 training and 44 test videos, a researcher can train an automatic affordance-hotspot labeling system and then use the generated labels to train an image segmentation model that recognizes functional parts during and outside moments of use.
- Because every task is recorded from both ego and frontal views, methods can learn from either perspective or exploit cross-view consistency between them.
- The twenty tasks cover affordances that involve multiple objects interacting with each other and the agent, such as Stack, Wrap, Filter, and Connect, going beyond the agent-object interactions covered by earlier video affordance datasets.
- The deliberate lack of prescribed preparation procedures means the same recordings can support research on preparatory maneuvers and space organization, informing collaborative robots that anticipate human actions.
Reading between the lines
- The absence of any reported annotation protocol, inter-annotator agreement, or baseline evaluation in the paper means the dataset should initially be read as a set of candidate labels; establishing benchmark quality would require an external reproducibility study.
- The a/b task variants with different object sets create a ready-made domain-shift test: a model trained on one variant can be evaluated on the other to check whether it learned the affordance itself rather than features of a particular object.
- At roughly seven hours of video, the dataset is small by current standards, but its dense annotation of demonstrations could serve as a seed set for bootstrapping larger automatically labeled video corpora for affordance learning.
- If the preparatory maneuvers observed in the recordings were later annotated systematically, the same videos would double as a resource for studying spatial arrangement and expected action sequences, not only affordance hotspots.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Wilhelm Tell Dataset, a collection of 189 videos (417 minutes) of human demonstrations of 20 household tasks, mainly centered on apples, recorded from both ego and third-person perspectives. The dataset includes task-level affordance labels, a train/test split, and for test videos CSV interval annotations and some still frames with affordance hotspot annotations. The authors state the intended use is to train and evaluate visual perception systems for affordance recognition, with an envisioned pipeline that relies on an external automatic hotspot labeling system. The paper describes the study setup, participant recruitment and consent, task lists, and usage notes, and the dataset is released under CC-BY 4.0.
Significance. If validated, the dataset would fill a notable gap by providing a video-based alternative to static-image affordance datasets, with multiple perspectives and an emphasis on affordances manifested through object-object and agent-object interactions. The inclusion of varied task performances and preparatory maneuvers could support research on task-space organization and collaborative robots. The paper also documents ethical procedures, GDPR-compliant storage, participant consent, and a public license, which are strengths. However, the contribution is currently unvalidated: the paper provides no annotation protocol, inter-annotator agreement measures, or baseline experiments, so the central usability claim is not yet supported.
major comments (4)
- [VI] The test-video CSV time intervals and hotspot annotations are the only human-verified labels in the dataset, but no annotation protocol is described and no inter-annotator agreement is reported. For example, for the Stab task it is unspecified whether the interaction interval starts at first contact, at full penetration, or at the lift of the fork. Without a precise semantic definition of an 'interaction' interval, annotations may be inconsistent across annotators and across sessions, undermining their validity as ground truth for the proposed evaluation pipeline.
- [VI] The paper claims the dataset is 'aimed towards training perception systems to recognize affordance manifestations' and 'can be used for robot training purposes,' yet no experiment or proof-of-concept demonstrates that the dataset actually supports such training. The envisioned use relies on an external automatic labeling system (e.g., [29]) that is not evaluated here. At least a baseline experiment, even a simple one, or quantitative evidence of annotation quality (e.g., coverage statistics, agreement) is needed to substantiate the usability claim.
- [V and VI] There is a numerical inconsistency in the dataset description: Section V reports 189 total videos, while Section VI states the videos are split into 102 training and 44 testing videos, which sums to 146. The paper does not explain how the remaining 43 videos, the combined per-perspective videos, and the incomplete participant entries relate to the train/test split. This ambiguity affects reproducibility and the interpretation of the dataset's composition.
- [IV and V] The mapping between tasks and affordance labels is not specified in a machine-readable schema. For instance, Task 4 in Table I is associated with two source labels (Contain-ability from AfNet and Contain from AffordanceNet), and it is unclear whether the CSV files contain only task identifiers or also the affordance labels derived from these sources. A clear metadata schema or example annotation file would significantly improve the dataset's usability.
minor comments (5)
- [II] The text contains several typos, including 'afordance' instead of 'affordance', 'demonstartion' instead of 'demonstration', and 'developement' instead of 'development'.
- [IV] There are typos in the methods section: 'headstreap' should be 'headstrap', 'corrent' should be 'correct', and 'layed out' should be 'laid out'.
- [V] The sentence 'The Readme file for the dataset containing information as well as a download link to the video material can be found on GitHub' is awkwardly phrased; consider rewriting for clarity and including the direct repository URL in the text.
- [VI] The word 'automtically' is a typo for 'automatically' in the description of the labeling pipeline.
- [IV] The explanation of the session ordering (tasks 8-10, then 1-7, then 11-20, with a/b variants) is somewhat confusing; a timeline or flow diagram of the bundles would help readers understand the recording structure.
Circularity Check
No significant circularity: the dataset paper derives no quantitative claims from its own inputs.
full rationale
This is a dataset introduction paper, not a derivation or modeling paper. It contains no equations, no fitted parameters, and no predicted benchmark numbers that could reduce to its own inputs by construction. The central contribution is the curated collection of 189 videos with task-level affordance labels, and the paper explicitly leaves training and evaluation to future users. The intended use case is stated as 'the training and evaluation of visual perception systems for affordance recognition,' which is a forward-looking goal rather than a result derived from the dataset. The affordance label set is taken from pre-existing external datasets (AfNet and AffordanceNet), with two added affordances chosen by the authors; this is an annotation design choice, not a circular derivation. The only self-citation among the authors, reference [17], appears in background context as a formal affordance model and is not load-bearing for the dataset's construction or claims. The lack of an annotation protocol or inter-annotator agreement for the test-video CSV intervals and hotspot annotations is a real validity and usability risk, but it is a data-quality concern, not circularity: the intervals are not being used to predict a quantity defined in terms of themselves. No self-definitional step, fitted-input-called-prediction step, or self-citation chain that forces the paper's conclusions was found. The honest finding is therefore a circularity score of 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Affordances can be learned by observing video demonstrations of the actions that manifest them.
- domain assumption The affordance labels in AfNet and AffordanceNet are an appropriate prior set of important affordances for robotics.
- domain assumption The head-mounted and frontal cameras capture enough visual information to recognize the target affordance manifestations.
- domain assumption The test-set CSV interval annotations and hotspot labels are accurate enough for training and evaluation.
Cite this review
Pith. "Pith review of The Wilhelm Tell Dataset of Affordance Demonstrations." pith.science (2026). https://pith.science/paper/U3KGUGLI
@misc{pith2026250717401,
author = {Pith},
title = {Pith review of: The Wilhelm Tell Dataset of Affordance Demonstrations},
year = {2026},
howpublished = {\url{https://pith.science/paper/U3KGUGLI}},
note = {Machine review of arXiv:2507.17401}
}
read the original abstract
Affordances - i.e. possibilities for action that an environment or objects in it provide - are important for robots operating in human environments to perceive. Existing approaches train such capabilities on annotated static images or shapes. This work presents a novel dataset for affordance learning of common household tasks. Unlike previous approaches, our dataset consists of video sequences demonstrating the tasks from first- and third-person perspectives, along with metadata about the affordances that are manifested in the task, and is aimed towards training perception systems to recognize affordance manifestations. The demonstrations were collected from several participants and in total record about seven hours of human activity. The variety of task performances also allows studying preparatory maneuvers that people may perform for a task, such as how they arrange their task space, which is also relevant for collaborative service robots.
Figures
Reference graph
Works this paper leans on
-
[29]
Multi-label affordance mapping from egocentric vision,
L. Mur-Labadia, J. J. Guerrero, and R. Martinez-Cantin, “Multi-label affordance mapping from egocentric vision,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 5238–5249
work page 2023
-
[1]
An interactive approach for situated task specification through verbal instructions,
C. Meric ¸li, S. D. Klee, J. Paparian, and M. Veloso, “An interactive approach for situated task specification through verbal instructions,” in Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems , 2014, pp. 1069–1076
work page 2014
-
[2]
Recent advances in robot learning from demonstration,
H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” Annual review of control, robotics, and autonomous systems , vol. 3, no. 1, pp. 297–330, 2020
2020
-
[3]
Describing upper-body motions based on labanotation for learning-from-observation robots,
K. Ikeuchi, Z. Ma, Z. Yan, S. Kudoh, and M. Nakamura, “Describing upper-body motions based on labanotation for learning-from-observation robots,” International Journal of Computer Vision , vol. 126, pp. 1415– 1429, 2018
work page 2018
-
[4]
V . N. Lu, J. Wirtz, W. H. Kunz, S. Paluch, T. Gruber, A. Martins, and P. G. Patterson, “Service robots, customers and service employees: what can we learn from the academic literature and where are the gaps?” Journal of Service Theory and Practice , vol. 30, no. 3, pp. 361–391, 2020
work page 2020
-
[5]
Droid: A large-scale in-the-wild robot manipulation dataset,
A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024
arXiv 2024
-
[6]
Prima-care: Privacy-preserving multi-modal dataset for human ac- tivity recognition in care robots,
A. Baselizadeh, M. Z. Uddin, W. Khaksar, D. S. Lindblom, and J. Torre- sen, “Prima-care: Privacy-preserving multi-modal dataset for human ac- tivity recognition in care robots,” in Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , 2024, pp. 233– 237
work page 2024
-
[7]
Y . Ishida and H. Tamukoh, “Semi-automatic dataset generation for object detection and recognition and its evaluation on domestic service robots,” Journal of Robotics and Mechatronics, vol. 32, no. 1, pp. 245–253, 2020
work page 2020
Show all 35 references
-
[8]
Object detection for sweeping robots in home scenes (odsr-ihs): a novel benchmark dataset,
Y . Lv, Y . Fang, W. Chi, G. Chen, and L. Sun, “Object detection for sweeping robots in home scenes (odsr-ihs): a novel benchmark dataset,” IEEE Access, vol. 9, pp. 17 820–17 828, 2021
2021
-
[9]
A multimodal dataset for object model learning from natural human-robot interaction,
P. Azagra, F. Golemo, Y . Mollard, M. Lopes, J. Civera, and A. C. Murillo, “A multimodal dataset for object model learning from natural human-robot interaction,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 6134–6141
2017
-
[10]
Air-act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,
W.-R. Ko, M. Jang, J. Lee, and J. Kim, “Air-act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,” The International Journal of Robotics Research , vol. 40, no. 4-5, pp. 691–697, 2021
2021
-
[11]
Robocup@ home: Evaluation of voice recognition systems for domestic service robots and introducing latino dataset,
E. Brinckhaus, G. T. Barnech, M. Etcheverry, and F. Andrade, “Robocup@ home: Evaluation of voice recognition systems for domestic service robots and introducing latino dataset,” in 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2...
2021
-
[12]
A dataset and system for service robot action interaction based on skeleton action recognition,
J. Wang, T. Zhang, X. Wu, and L. Zeng, “A dataset and system for service robot action interaction based on skeleton action recognition,” in 2023 8th International Conference on Signal and Image Processing (ICSIP). IEEE, 2023, pp. 41–48
2023
-
[13]
Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly,
J. Jang, D. Kim, C. Park, M. Jang, J. Lee, and J. Kim, “Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 10 990–10 997
2020
-
[14]
Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,
H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 807–11 814, 2022
2022
-
[15]
Ssup-hri: Social signaling in urban public human-robot interaction dataset,
F. Bu and W. Ju, “Ssup-hri: Social signaling in urban public human-robot interaction dataset,” arXiv preprint arXiv:2403.10994 , 2024
2024 arXiv
-
[16]
J. J. Gibson, The Ecological Approach to Visual Perception. Psychology Press Classic Editions, 1979
1979
-
[17]
A formal model of affordances for flexible robotic task execution,
D. Beßler, R. Porzel, M. Pomarlan, M. Beetz, R. Malaka, and J. Bate- man, “A formal model of affordances for flexible robotic task execution,” in Proceedings of the 24th European Conference on Artificial Intelli- gence (ECAI), 09 2020
2020
-
[18]
Affordance ontology: towards a unified description of affordances as events,
L. A. Moralez, “Affordance ontology: towards a unified description of affordances as events,” Res. Cogitans, vol. 7, no. 1, pp. 35–45, 2016
2016
-
[19]
Modeling affordances with dispositions,
F. TOYOSHIMA, “Modeling affordances with dispositions,” in Proceed- ings of the Joint Ontology Workshops , 2018
2018
-
[20]
Affordances, motivation, and the design of user interfaces,
J. Karat, C.-m. Karat, and J. Ukelson, “Affordances, motivation, and the design of user interfaces,” Commun. ACM, vol. 43, pp. 49–51, 08 2000
2000
-
[21]
A review of affordances and affordance-based design to address usability,
N. Masoudi, G. M. Fadel, C. C. Pagano, and M. V . Elena, “A review of affordances and affordance-based design to address usability,” Proceed- ings of the Design Society: International Conference on Engineering Design, vol. 1, no. 1, p. 1353–1362, 2019
2019
-
[22]
Visual affordance and function understanding: A survey,
M. Hassanin, S. Khan, and M. Tahtali, “Visual affordance and function understanding: A survey,” ACM Comput. Surv. , vol. 54, no. 3, Apr
-
[23]
Afnet: The affordance network,
K. M. Varadarajan and M. Vincze, “Afnet: The affordance network,” in Asian Conference on Computer Vision , 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:39914795
2012
-
[24]
3d affordancenet: A benchmark for visual object affordance understanding,
S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia, “3d affordancenet: A benchmark for visual object affordance understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2021
2021
-
[25]
Open-vocabulary affordance detection in 3d point clouds,
T. Nguyen, M. N. Vu, A. Vuong, D. Nguyen, T. V o, N. Le, and A. Nguyen, “Open-vocabulary affordance detection in 3d point clouds,” in Proceedings of the International Conference on Robotic Systems (IROS), 2023
2023
-
[26]
Learning affordance grounding from exocentric images,
H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning affordance grounding from exocentric images,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2242–2251, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:247594536
2022
-
[27]
Cerberus transformer: Joint semantic, affordance and attribute parsing,
X. Chen, T. Liu, H. Zhao, G. Zhou, and Y .-Q. Zhang, “Cerberus transformer: Joint semantic, affordance and attribute parsing,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 19 617–19 626, 2021. [Online]. Available: https://api.semanticscholar...
2022
-
[28]
Openscene: 3d scene understanding with open vocabularies,
S. Peng, K. Genova, ChiyuMaxJiang, A. Tagliasacchi, M. Pollefeys, and T. A. Funkhouser, “Openscene: 3d scene understanding with open vocabularies,” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 815–824, 2022. [Online]. Available: https://api....
2023
-
[30]
Learning visual affordance grounding from demonstration videos,
H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning visual affordance grounding from demonstration videos,” IEEE Transactions on Neural Networks and Learning Systems , 2023
2023
-
[31]
Demo2vec: Reasoning object affordances from online videos,
K. Fang, T.-L. Wu, D. Yang, S. Savarese, and J. J. Lim, “Demo2vec: Reasoning object affordances from online videos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 2139–2147
2018
-
[32]
The epic- kitchens dataset: Collection, challenges and baselines,
D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kaza- kos, D. Moltisanti, J. Munro, T. Perrett, W. Price et al. , “The epic- kitchens dataset: Collection, challenges and baselines,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence , vol. 43, no...
2020
-
[33]
Ego4d: Around the world in 3,000 hours of egocentric video,
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 1...
2022
-
[34]
Affordances from human videos as a versatile representation for robotics,
S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 778–13 790
2023
- [2021]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.